Creation and user interactions with three-dimensional wallpaper on computing devices

ABSTRACT

A wallpaper system presents a first wallpaper image of a wallpaper video to a user and receives, via a user input device, one or both of: (i) a spatial user input selection, and (ii) a time user input selection from the user to apply to the wallpaper video. In response to detecting one or both of: (i) the spatial user input selection, and (ii) the time user input selection, the wallpaper system determines one or both of: (i) a respective spatial movement parameter within a wallpaper video associated with the spatial user input selection, and (ii) a respective time coordinate within the wallpaper video associated with the time user input selection. Wallpaper system presents, via the image display, a second wallpaper image associated with one or both of: (i) the respective spatial movement parameter, and (ii) the respective time coordinate.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional application Ser.No. 62/784,914 entitled CREATION AND USER INTERACTIONS WITHTHREE-DIMENSIONAL WALLPAPER ON COMPUTING DEVICES, filed on Dec. 26,2018, the contents of which are incorporated fully herein by reference.

TECHNICAL FIELD

The present subject matter relates to wearable devices, e.g., eyeweardevices, and mobile devices and techniques to create and allow a user tointeract with three-dimensional wallpaper videos and images.

BACKGROUND

A wallpaper or background (e.g., a desktop wallpaper, desktopbackground, desktop picture or desktop image on computers) is atwo-dimensional (2D) digital image (photo, drawing etc.) used as adecorative background of a graphical user interface on the screen of acomputer, mobile communications device or other electronic device. On acomputer, it is usually for the desktop, while on a mobile phone it isusually the background for the “home” or “idle” screen. Though mostdevices come with a default picture, users can usually change it tocustom files of their choosing.

Computing devices, such as wearable devices, including portable eyeweardevices (e.g., smartglasses, headwear, and headgear); mobile devices(e.g., tablets, smartphones, and laptops); and personal computersavailable today integrate image displays and cameras. Currently, usersof computing devices can utilize photo lenses or filters to createeffects on two-dimensional (2D) photographs. Various photo decoratingapplications feature tools like stickers, emojis, and captions to edittwo-dimensional photographs.

With the advent of three-dimensional (3D) image and video content, moresophisticated manipulations and interactions to transformthree-dimensional image and video content (e.g., videos, pictures, etc.)are needed. For example, being able to manipulate and interact with thethree-dimensional image and video content to create graphical effects onthree-dimensional images and videos is desirable. Thus far, userinteractions with wallpaper have been very limited. Accordingly, a needexists to enhance video and image graphical effects available forthree-dimensional image and video wallpaper content to allow for moreinteresting user interaction experiences.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawing figures depict one or more implementations, by way ofexample only, not by way of limitations. In the figures, like referencenumerals refer to the same or similar elements.

FIG. 1A is a right side view of an example hardware configuration of aneyewear device utilized in a wallpaper system, in which one or both of:(i) a spatial user input selection, and (ii) a time user input selectionfrom a user are applied to interact with wallpaper images.

FIG. 1B is a top cross-sectional view of a right chunk of the eyeweardevice of FIG. 1A depicting a right visible light camera of adepth-capturing camera, and a circuit board.

FIG. 1C is a left side view of an example hardware configuration of aneyewear device of FIG. 1A, which shows a left visible light camera ofthe depth-capturing camera.

FIG. 1D is a top cross-sectional view of a left chunk of the eyeweardevice of FIG. 1C depicting the left visible light camera of thedepth-capturing camera, and the circuit board.

FIG. 2A is a right side view of another example hardware configurationof an eyewear device utilized in the wallpaper system, which shows theright visible light camera and a depth sensor of the depth-capturingcamera to generate a depth image.

FIGS. 2B and 2C are rear views of example hardware configurations of theeyewear device, including two different types of image displays.

FIG. 3 shows a rear perspective sectional view of the eyewear device ofFIG. 2A depicting an infrared camera of the depth sensor, a frame front,a frame back, and a circuit board.

FIG. 4 is a cross-sectional view taken through the infrared camera andthe frame of the eyewear device of FIG. 3.

FIG. 5 shows a rear perspective view of the eyewear device of FIG. 2Adepicting an infrared emitter of the depth sensor, the infrared cameraof the depth sensor, the frame front, the frame back, and the circuitboard.

FIG. 6 is a cross-sectional view taken through the infrared emitter andthe frame of the eyewear device of FIG. 5.

FIG. 7 depicts an example of a pattern of infrared light emitted by theinfrared emitter of the depth sensor and reflection variations of theemitted pattern of infrared light captured by the infrared camera of thedepth sensor of the eyewear device to measure depth of pixels in a rawimage to generate the depth image.

FIG. 8A depicts an example of infrared light captured by the infraredcamera of the depth sensor as an infrared image and visible lightcaptured by a visible light camera as a raw image to generate theinitial depth image of a three-dimensional scene.

FIG. 8B depicts an example of visible light captured by the left visiblelight camera as left raw image and visible light captured by the rightvisible light camera as a right raw image to generate the initial depthimage of a three-dimensional scene.

FIG. 9 is a high-level functional block diagram of an example wallpapersystem including the eyewear device with a depth-capturing camera tocapture an original video that include original images, a mobile device,and a server system connected via various networks.

FIG. 10 shows an example of a hardware configuration for the mobiledevice of the wallpaper system of FIG. 9 to create and interact with awallpaper video that includes multiple wallpaper images.

FIG. 11 is a flowchart of a method that can be implemented in thewallpaper system to apply to the original video to create and present awallpaper image.

FIG. 12 illustrates an example of a first original image associated witha first time coordinate of an original video, which is a processed(e.g., rectified) image captured by one or both of the visible lightcameras.

FIG. 13 illustrates receiving, via a movement tracker type of user inputdevice, a first spatial user input selection (e.g., horizontal tiltingto the left) to manipulate a spatial movement parameter (e.g., field ofview) and responsively presenting a first wallpaper image.

FIG. 14 illustrates receiving, via the movement tracker, a secondspatial user input selection (e.g., horizontal tilting to the right) andresponsively presenting a second wallpaper image.

FIG. 15 illustrates receiving, via the movement tracker, a third spatialuser input selection (e.g., horizontal tilting to the left) and againresponsively presenting the first wallpaper image.

FIG. 16 again illustrates the first wallpaper image of FIG. 15 anddepicts initiating of finger contact, via a touch sensor type of userinput device, to manipulate a time coordinate (e.g., temporalcoordinate).

FIG. 17 illustrates receiving, via the touch sensor, a first time userinput selection (e.g., vertical scrolling upwards) and responsivelypresenting the first wallpaper image associated with a second timecoordinate.

FIG. 18 illustrates receiving, via the touch sensor, a second time userinput selection (e.g., vertical scrolling upwards) and responsivelypresenting the first wallpaper image associated with a third timecoordinate.

FIG. 19 illustrates receiving, via the touch sensor, a third time userinput selection (e.g., vertical scrolling upwards) and responsivelypresenting the first wallpaper image associated with a fourth timecoordinate.

FIG. 20 illustrates receiving, via the touch sensor, a fourth time userinput selection (e.g., vertical scrolling upwards) and responsivelypresenting the first wallpaper image associated with a fifth timecoordinate.

FIG. 21 illustrates receiving, via the touch sensor, a fifth time userinput selection (e.g., vertical scrolling downwards) and againresponsively presenting the first wallpaper image associated with thefourth time coordinate of FIG. 19.

FIG. 22 illustrates receiving, via the touch sensor, a sixth time userinput selection (e.g., vertical scrolling downwards) and againresponsively presenting the first wallpaper image associated with thefirst time coordinate of FIGS. 15-16.

FIG. 23 illustrates wallpaper selection on the mobile device, includingvarious types of original videos and original images that are selectableas wallpaper.

FIG. 24 illustrates selection of an original image as wallpaper on themobile device.

FIG. 25 illustrates setting of the original image of FIG. 24 aswallpaper on the mobile device.

FIG. 26 illustrates the original image of FIG. 25, which is a processed(e.g., rectified) image captured by one or both of the visible lightcameras.

FIG. 27 illustrates receiving, via the movement tracker, a first spatialuser input selection (e.g., horizontal tilting to the left) tomanipulate a spatial movement parameter and responsively presenting afirst wallpaper image associated with a first spatial movementparameter.

FIG. 28 illustrates receiving, via the movement tracker, a secondspatial user input selection (e.g., horizontal tilting to the right) andresponsively presenting a second wallpaper image associated with asecond spatial movement parameter.

FIG. 29 again illustrates wallpaper selection on the mobile device,including various types of original videos and original images that areselectable as wallpaper.

FIG. 30 illustrates selection of an original video as wallpaper on themobile device.

FIG. 31 illustrates setting of the original video of FIG. 30 aswallpaper on the mobile device.

FIG. 32 illustrates an example of a second original image, which is aprocessed (e.g., rectified) image associated with a second timecoordinate of the original video of FIG. 30.

FIG. 33 illustrates receiving, via the touch sensor, a first time userinput selection (e.g., vertical scrolling downwards) to manipulate atime coordinate and responsively presenting a sixth wallpaper imageassociated with a first time coordinate.

FIG. 34 illustrates receiving, via the touch sensor, a second time userinput selection (e.g., vertical scrolling upwards) and responsivelypresenting the sixth wallpaper image associated with a third timecoordinate.

FIG. 35 illustrates receiving, via the touch sensor, a third time userinput selection (e.g., vertical scrolling upwards) and responsivelypresenting the sixth wallpaper image associated with a fourth timecoordinate.

FIG. 36 illustrates receiving, via the movement tracker, a first spatialuser input selection (e.g., horizontal tilting to the left) andresponsively presenting the first wallpaper image associated with thefourth time coordinate of the fourth original image.

FIG. 37 illustrates receiving, via the movement tracker, a secondspatial user input selection (e.g., horizontal tilting to the right) andresponsively presenting the eighth wallpaper image associated with thefourth time coordinate of the fourth original image.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth by way of examples in order to provide a thorough understanding ofthe relevant teachings. However, it should be apparent to those skilledin the art that the present teachings may be practiced without suchdetails. In other instances, description of well-known methods,procedures, components, and circuitry are set forth at a relativelyhigh-level, without detail, in order to avoid unnecessarily obscuringaspects of the present teachings.

As used herein, the term “wallpaper” means a computer generated effectapplied to an original image or sequence of original images thatmanipulates a spatial movement parameter of the original image to changea field of view or change a time coordinate of the sequence of originalimages. As described herein, a first way to manipulate the spatialmovement parameter is via a “light field effect” and a second way tomanipulate the spatial movement parameter is processing inthree-dimensional space to create a depth image (e.g., a mesh ofvertices, which are texture mapped) by rotating through the mesh ofvertices.

Generally, the term “light field” means radiance at a point in a givendirection. The term “light field effect” means rendering a differentview of a scene of image(s) to provide an appearance of spatial movementor rotation as if the observer is viewing the scene from a differentangle or perspective. Light field effect cameras can capture light fromdifferent directions and move around to create a scene in three or fourdimensions (e.g., using multiple lenses). However, such processing inthree-dimensional (X, Y, and Z) and four-dimensional space (X, Y, Z, andtime) is relatively complex and can be computationally intensive. Twovisible light cameras 114A-B can be used to create a simplified lightfield effect from two images by operating in two-dimensional space only,which is less computationally intensive.

The term “coupled” or “connected” as used herein refers to any logical,optical, physical or electrical connection, link or the like by whichelectrical or magnetic signals produced or supplied by one systemelement are imparted to another coupled or connected element. Unlessdescribed otherwise, coupled or connected elements or devices are notnecessarily directly connected to one another and may be separated byintermediate components, elements or communication media that maymodify, manipulate or carry the electrical signals. The term “on” meansdirectly supported by an element or indirectly supported by the elementthrough another element integrated into or supported by the element.

The orientations of the eyewear device, associated components and anycomplete devices incorporating a depth-capturing camera such as shown inany of the drawings, are given by way of example only, for illustrationand discussion purposes. In operation for wallpaper creation and userinteraction, the eyewear device may be oriented in any other directionsuitable to the particular application of the eyewear device, forexample up, down, sideways, or any other orientation. Also, to theextent used herein, any directional term, such as front, rear, inwards,outwards, towards, left, right, lateral, longitudinal, up, down, upper,lower, top, bottom, side, horizontal, vertical, and diagonal are used byway of example only, and are not limiting as to direction or orientationof any depth-capturing camera or component of the depth-capturing cameraconstructed as otherwise described herein.

Additional objects, advantages and novel features of the examples willbe set forth in part in the following description, and in part willbecome apparent to those skilled in the art upon examination of thefollowing and the accompanying drawings or may be learned by productionor operation of the examples. The objects and advantages of the presentsubject matter may be realized and attained by means of themethodologies, instrumentalities and combinations particularly pointedout in the appended claims.

Reference now is made in detail to the examples illustrated in theaccompanying drawings and discussed below.

FIG. 1A is a right side view of an example hardware configuration of aneyewear device 100 utilized in a wallpaper system, which shows a rightvisible light camera 114B of a depth-capturing camera to generate adepth image. As further described below, in the wallpaper system, one orboth of: (i) a spatial user input selection, and (ii) a time user inputselection are received from a user. To present a high degree of userinteraction, the received spatial and time user input selections areapplied to an original image or an original video that includes asequence of original images selected as wallpaper. For example, thewallpaper system can vary one or both of: (i) the spatial movementparameter (e.g., field of view) of the original images (e.g. a raw imageor a processed raw image), and (ii) the time coordinate of the originalvideo. Both two-dimensional (2D) techniques, such as light fieldeffects, and three-dimensional (3D) techniques, such as depthprocessing, can be applied to vary the spatial movement parameter. Inone example, the spatial user input selection and the time user inputselection can be received via a user input device, such as a movementtracker (e.g., accelerometer, gyroscope, or inertial measurement unit)or a touch sensor of a mobile device. Tilting the mobile device left orright manipulates the spatial movement parameter of the original imageand scrolling upwards or downwards on the touch sensor manipulates thetime coordinate to move the original video forward or backwards in time.

Eyewear device 100, includes a right optical assembly 180B with an imagedisplay to present images, such as an original image (e.g., based on aleft raw image, a processed left image, a right raw image, or aprocessed right image) and a wallpaper image. As shown in FIGS. 1A-B,the eyewear device 100 includes the right visible light camera 114B.Eyewear device 100 can include multiple visible light cameras 114A-Bthat form a passive type of depth-capturing camera, such as stereocamera, of which the right visible light camera 114B is located on aright chunk 110B. As shown in FIGS. 1C-D, the eyewear device 100 canalso include a left visible light camera 114A. Alternatively, in theexample of FIG. 2A, the depth-capturing camera can be an active type ofdepth-capturing camera that includes a single visible light camera 114Aand a depth sensor (see element 213 of FIG. 2A).

Left and right visible light cameras 114A-B are sensitive to the visiblelight range wavelength. Each of the visible light cameras 114A-B have adifferent frontward facing field of view which are overlapping to allowthree-dimensional depth images to be generated, for example, rightvisible light camera 114B has the depicted right field of view 111B.Generally, a “field of view” is the part of the scene that is visiblethrough the camera at a particular position and orientation in space.Objects or object features outside the field of view 111A-B when theimage is captured by the visible light camera are not recorded in a rawimage (e.g., photograph or picture). The field of view describes anangle range or extent which the image sensor of the visible light camera114A-B picks up electromagnetic radiation of a given scene in a capturedimage of the given scene. Field of view can be expressed as the angularsize of the view cone, i.e., an angle of view. The angle of view can bemeasured horizontally, vertically, or diagonally.

In an example, visible light cameras 114A-B have a field of view with anangle of view between 15° to 30°, for example 24°, and have a resolutionof 480×480 pixels. The “angle of coverage” describes the angle rangethat a lens of visible light cameras 114A-B or infrared camera 220 (seeFIG. 2A) can effectively image. Typically, the image circle produced bya camera lens is large enough to cover the film or sensor completely,possibly including some vignetting toward the edge. If the angle ofcoverage of the camera lens does not fill the sensor, the image circlewill be visible, typically with strong vignetting toward the edge, andthe effective angle of view will be limited to the angle of coverage.

Examples of such visible lights camera 114A-B include a high-resolutioncomplementary metal-oxide-semiconductor (CMOS) image sensor and a videographic array (VGA) camera, such as 640p (e.g., 640×480 pixels for atotal of 0.3 m 3egapixels), 720p, or 1080p. As used herein, the term“overlapping” when referring to field of view means the matrix of pixelsin the generated raw image(s) or infrared image of a scene overlap by30% or more. As used herein, the term “substantially overlapping” whenreferring to field of view means the matrix of pixels in the generatedraw image(s) or infrared image of a scene overlap by 50% or more.

Image sensor data from the visible light cameras 114A-B are capturedalong with geolocation data, digitized by an image processor, and storedin a memory. The captured left and right raw images captured byrespective visible light cameras 114A-B are in the two-dimensional spacedomain and comprise a matrix of pixels on a two-dimensional coordinatesystem that includes an X axis for horizontal position and a Y axis forvertical position. Each pixel includes a color attribute (e.g., a redpixel light value, a green pixel light value, and/or a blue pixel lightvalue); and a position attribute (e.g., an X location coordinate and a Ylocation coordinate).

To provide stereoscopic vision, visible light cameras 114A-B may becoupled to an image processor (element 912 of FIG. 9) for digitalprocessing along with a timestamp in which the image of the scene iscaptured. Image processor 912 includes circuitry to receive signals fromthe visible light cameras 114A-B and process those signals from thevisible light camera 114 into a format suitable for storage in thememory. The timestamp can be added by the image processor or otherprocessor, which controls operation of the visible light cameras 114A-B.Visible light cameras 114A-B allow the depth-capturing camera tosimulate human binocular vision. Depth-capturing camera provides theability to reproduce three-dimensional images based on two capturedimages from the visible light cameras 114A-B having the same timestamp.Such three-dimensional images allow for an immersive life-likeexperience, e.g., for virtual reality or video gaming.

Rectification is applied so that each captured image or video ismodified so that corresponding pixels lie on the same raster line (row).Once this is done, the image disparity computation algorithm, such asSemi-Global Block Matching (SGBM) is applied. The disparity computationalgorithm finds a corresponding pixel for each pixel in the left imagein the right image. And for each pixel in the right image, finds acorresponding pixel in the left image. Usually the same disparity isfound from left to right and right to left for non-occluded pixels(pixels seen from both cameras); however, occluded pixels are treatedseparately, typically by neighbor pixel blending techniques.

For stereoscopic vision, a pair of raw red, green, and blue (RGB) imagesare captured of a scene at a given moment in time—one image for each ofthe left and right visible light cameras 114A-B (e.g., stereo pairs).When the pair of captured raw images from the frontward facing left andright field of views 111A-B of the left and right visible light cameras114A-B are processed (e.g., by the image processor), depth images aregenerated. Depth images can be based on a three-dimensional model thatcan include a three-dimensional mesh (e.g., triangulated mesh) andtextures, which are uploaded to a graphics processing unit (GPU) asvertices along with texture mapping. Usually, the depth is not actuallyseen, but the effect of depth can be seen in the rendered and displayedtwo-dimensional images. The generated depth images can be transformed tobe perceived by a user on the optical assembly 180A-B or other imagedisplay(s) (e.g., of a mobile device) by transforming those depth imagesinto various viewpoints that are two-dimensional images for display. Thegenerated depth images are in the three-dimensional space domain and cancomprise a mesh of vertices on a three-dimensional location coordinatesystem that includes an X axis for horizontal position (e.g., length), aY axis for vertical position (e.g., height), and a Z axis for depth(e.g., distance). Each vertex includes a position attribute (e.g., a redpixel light value, a green pixel light value, and/or a blue pixel lightvalue); a position attribute (e.g., an X location coordinate, a Ylocation coordinate, and a Z location coordinate); a texture attribute,and/or a reflectance attribute. The texture attribute quantifies theperceived texture of the depth image, such as the spatial arrangement ofcolor or intensities in a region of vertices of the depth image.

Generally, perception of depth arises from the disparity of a given 3Dpoint in the left and right raw images captured by visible light cameras114A-B. Disparity is the difference in image location of the same 3Dpoint when projected under perspective of the visible light cameras114A-B (d=x_(left)−x_(right)). Correlation of the left and right pixelsin the respective left and right raw images can be achieved withSemi-Global Block Matching (SGBM), for example. For visible lightcameras 114A-B with parallel optical axes, focal length f, baseline b,and corresponding image points (x_(left), y_(left)) and (x_(right),y_(right)), the location of a 3D point (Z axis location coordinate) canbe derived utilizing triangulation which determines depth fromdisparity. Typically, depth of the 3D point is inversely proportional todisparity. A variety of other techniques can also be used. Generation ofthree-dimensional depth images and wallpaper images is explained in moredetail later.

In an example, a wallpaper system includes the eyewear device 100. Theeyewear device 100 includes a frame 105 and a left temple 110A extendingfrom a left lateral side 170A of the frame 105 and a right temple 110Bextending from a right lateral side 170B of the frame 105. Eyeweardevice 100 further includes a depth-capturing camera. Thedepth-capturing camera includes: (i) at least two visible light cameraswith overlapping fields of view; or (ii) a least one visible lightcamera 114A-B and a depth sensor (element 213 of FIG. 2A). In oneexample, the depth-capturing camera includes a left visible light camera114A with a left field of view 111A connected to the frame 105 or theleft temple 110A to capture a left image of the scene. Eyewear device100 further includes a right visible light camera 114B connected to theframe 105 or the right temple 110B with a right field of view 111B tocapture (e.g., simultaneously with the left visible light camera 114A) aright image of the scene which partially overlaps the left image.

Wallpaper system further includes a computing device, such as a hostcomputer (e.g., mobile device 990 of FIGS. 9-10) coupled to eyeweardevice 100 over a network. The wallpaper system further includes animage display (optical assembly 180A-B of eyewear device; image display1080 of mobile device 990 of FIG. 10) for presenting (e.g., displaying)a sequence of images. The sequence of images includes the originalimages, raw images or processed raw images in two-dimensional space(e.g., after rectification), and wallpaper images. Wallpaper systemfurther includes an image display driver (element 942 of eyewear device100 of FIG. 9; element 1090 of mobile device 990 of FIG. 10) coupled tothe image display (optical assembly 180A-B of eyewear device; imagedisplay 1080 of mobile device 990 of FIG. 10) to control the imagedisplay to present the sequence of images. The sequence of images caninclude the original images, such as the raw images or processed rawimages in two-dimensional space (e.g., after rectification), andwallpaper images.

Wallpaper system further includes at least one user input device toreceive one or both of: (i) a spatial user input selection, and (ii) atime user input selection. Examples of user input devices include atouch sensor (element 991 of FIG. 9 for the eyewear device 100), a touchscreen display (element 1091 of FIG. 10 for the mobile device 1090).User input devices also include a movement tracker such as anaccelerometer, gyroscope, and inertial measurement unit (element 981 inFIGS. 9-10 for both the eyewear device and the mobile device 1090); anda computer mouse for a personal computer or a laptop computer. Wallpapersystem further includes a processor (element 932 of eyewear device 100of FIG. 9; element 1030 of mobile device 990 of FIG. 10) coupled to theeyewear device 100 and the depth-capturing camera. Wallpaper systemfurther includes a memory (element 934 of eyewear device 100 of FIG. 9;elements 1040A-B of mobile device 990 of FIG. 10) accessible to theprocessor, and wallpaper programming in the memory (element 945 ofeyewear device 100 of FIG. 9; element 945 of mobile device 990 of FIG.10), for example in the eyewear device 100 itself, mobile device(element 990 of FIG. 9), or another part of the wallpaper system (e.g.,server system 998 of FIG. 9).

As explained below to provide spatial movement by applying 2D processingusing light field effects, the wallpaper system takes a left image and aright image as input viewpoints, but no images with viewpoints inbetween. To generate the light field effect, where a character jumps andthe camera rotates around the character at different angles as thatmoment is frozen in time, interpolation is performed between the leftand right images captured by the left and right cameras 114A-B. Lightfield effect images from several different viewpoints can be stitchedtogether as a sequence of images in a video to provide spatial movement.For example, the spatial movement parameter may vary between 0 and 1 inincrements of 0.1, for a total of eleven viewpoints based on, forexample, a tilt angle of a mobile device. When the spatial movementparameter is set to 0.0, the field of view skews entirely to the leftcamera perspective. When the spatial movement parameter is set to 0.5,the field of view is in the middle of the left and right cameraperspective. When the spatial movement parameter is set to 1.0, thefield of view skews entirely to the right visible light cameraperspective.

In a 2D image processing implementation, two left and right images areinterpolated to generate the wallpaper image and the interpolation isbased on the disparity maps generated from the two original RGB images.This provides an appearance of a 3D world sensation by rotating imagesthat are not even real, but only requires two modified two-dimensionalimages (frames) to produce the light field effect. Disparity mapsdetermine how many pixels to move between pixels in the left image toobtain a corresponding pixel in the right image, and vice versa.Disparity is calculated between a stereo pair of corresponding pixels,which corresponds to depth, in order to interpolate between two imagesand blend the left and right images together to provide an appearance ofrotation or movement in the created wallpaper image.

FIG. 1B is a top cross-sectional view of a right chunk 110B of theeyewear device 100 of FIG. 1A depicting the right visible light camera114B of the depth-capturing camera, and a circuit board 140B. FIG. 1C isa left side view of an example hardware configuration of an eyeweardevice 100 of FIG. 1A, which shows a left visible light camera 114A ofthe depth-capturing camera. FIG. 1D is a top cross-sectional view of aleft chunk 110A of the eyewear device of FIG. 1C depicting the leftvisible light camera 114A of the depth-capturing camera, and a circuitboard 140A. Construction and placement of the left visible light camera114A is substantially similar to the right visible light camera 114B,except the connections and coupling are on the left lateral side 170A.As shown in the example of FIG. 1B, the eyewear device 100 includes theright visible light camera 114B and a circuit board, which may be aflexible printed circuit board (PCB) 140B. The right hinge 226B connectsthe right chunk 110B to a right temple 125B of the eyewear device 100.Similarly, the left hinge 226A connects the left chunk 110A to a lefttemple 125A of the eyewear device 100. In some examples, components ofthe right visible light camera 114B, the flexible PCB 140B, or otherelectrical connectors or contacts may be located on the right temple125B or the right hinge 226B.

The right chunk 110B includes chunk body 211 and a chunk cap, with thechunk cap omitted in the cross-section of FIG. 1B. Disposed inside theright chunk 110B are various interconnected circuit boards, such as PCBsor flexible PCBs, that include controller circuits for right visiblelight camera 114B, microphone(s) 130, speaker(s) 132, low-power wirelesscircuitry (e.g., for wireless short range network communication viaBluetooth™), high-speed wireless circuitry (e.g., for wireless localarea network communication via WiFi).

The right visible light camera 114B is coupled to or disposed on theflexible PCB 140B and covered by a visible light camera cover lens,which is aimed through opening(s) formed in the frame 105. For example,the right rim 107B of the frame 105 is connected to the right chunk 110Band includes the opening(s) for the visible light camera cover lens. Theframe 105 includes a front-facing side configured to face outwards awayfrom the eye of the user. The opening for the visible light camera coverlens is formed on and through the front-facing side. In the example, theright visible light camera 114B has an outward facing field of view 111Bwith a line of sight or perspective of the right eye of the user of theeyewear device 100. The visible light camera cover lens can also beadhered to an outward facing surface of the right chunk 110B in which anopening is formed with an outward facing angle of coverage, but in adifferent outwards direction. The coupling can also be indirect viaintervening components.

Left (first) visible light camera 114A is connected to a left imagedisplay of left optical assembly 180A to capture a left eye viewed sceneobserved by a wearer of the eyewear device 100 in a left raw image.Right (second) visible light camera 114B is connected to a right imagedisplay of right optical assembly 180B to capture a right eye viewedscene observed by the wearer of the eyewear device 100 in a right rawimage. The left raw image and the right raw image partially overlap topresent a three-dimensional observable space of a generated depth image.

Flexible PCB 140B is disposed inside the right chunk 110B and is coupledto one or more other components housed in the right chunk 110B. Althoughshown as being formed on the circuit boards of the right chunk 110B, theright visible light camera 114B can be formed on the circuit boards ofthe left chunk 110A, the temples 125A-B, or frame 105.

FIG. 2A is a right side view of another example hardware configurationof an eyewear device 100 utilized in the wallpaper system. As shown, thedepth-capturing camera includes a left visible light camera 114A and adepth sensor 213 on a frame 105 to generate a depth image. Instead ofutilizing at least two visible light cameras 114A-B to generate thedepth image, here a single visible light camera 114A and the depthsensor 213 are utilized to generate depth images, such as the depthimage. As in the example of FIGS. 1A-D, a spatial user input selectionfrom a user can be applied to a depth image, which is a 3D model of anoriginal image, to create a wallpaper image based on the spatial userinput selection, and then present the wallpaper image. The infraredcamera 220 of the depth sensor 213 has an outward facing field of viewthat substantially overlaps with the left visible light camera 114A fora line of sight of the eye of the user. As shown, the infrared emitter215 and the infrared camera 220 are co-located on the upper portion ofthe left rim 107A with the left visible light camera 114A.

In the example of FIG. 2A, the depth sensor 213 of the eyewear device100 includes an infrared emitter 215 and an infrared camera 220 whichcaptures an infrared image. Visible light cameras 114A-B typicallyinclude a blue light filter to block infrared light detection, in anexample, the infrared camera 220 is a visible light camera, such as alow resolution video graphic array (VGA) camera (e.g., 640×480 pixelsfor a total of 0.3 megapixels), with the blue filter removed. Theinfrared emitter 215 and the infrared camera 220 are co-located on theframe 105, for example, both are shown as connected to the upper portionof the left rim 107A. As described in further detail below, the frame105 or one or more of the left and right chunks 110A-B include a circuitboard that includes the infrared emitter 215 and the infrared camera220. The infrared emitter 215 and the infrared camera 220 can beconnected to the circuit board by soldering, for example.

Other arrangements of the infrared emitter 215 and infrared camera 220can be implemented, including arrangements in which the infrared emitter215 and infrared camera 220 are both on the right rim 107B, or indifferent locations on the frame 105, for example, the infrared emitter215 is on the left rim 107A and the infrared camera 220 is on the rightrim 107B. However, the at least one visible light camera 114A and thedepth sensor 213 typically have substantially overlapping fields of viewto generate three-dimensional depth images. In another example, theinfrared emitter 215 is on the frame 105 and the infrared camera 220 ison one of the chunks 110A-B, or vice versa. The infrared emitter 215 canbe connected essentially anywhere on the frame 105, left chunk 110A, orright chunk 110B to emit a pattern of infrared in the light of sight ofthe eye of the user. Similarly, the infrared camera 220 can be connectedessentially anywhere on the frame 105, left chunk 110A, or right chunk110B to capture at least one reflection variation in the emitted patternof infrared light of a three-dimensional scene in the light of sight ofthe eye of the user.

The infrared emitter 215 and infrared camera 220 are arranged to faceoutwards to pick up an infrared image of a scene with objects or objectfeatures that the user wearing the eyewear device 100 observes. Forexample, the infrared emitter 215 and infrared camera 220 are positioneddirectly in front of the eye, in the upper part of the frame 105 or inthe chunks 110A-B at either ends of the frame 105 with a forward facingfield of view to capture images of the scene which the user is gazingat, for measurement of depth of objects and object features.

In one example, the infrared emitter 215 of the depth sensor 213 emitsinfrared light illumination in the forward facing field of view of thescene, which can be near-infrared light or other short-wavelength beamof low-energy radiation. Alternatively, or additionally, the depthsensor 213 may include an emitter that emits other wavelengths of lightbesides infrared and the depth sensor 213 further includes a camerasensitive to that wavelength that receives and captures images with thatwavelength. As noted above, the eyewear device 100 is coupled to aprocessor and a memory, for example in the eyewear device 100 itself oranother part of the wallpaper system. Eyewear device 100 or thewallpaper system can subsequently process the captured infrared imageduring generation of three-dimensional depth images, such as the depthimage.

FIGS. 2B-C are rear views of example hardware configurations of theeyewear device 100, including two different types of image displays.Eyewear device 100 is in a form configured for wearing by a user, whichare eyeglasses in the example. The eyewear device 100 can take otherforms and may incorporate other types of frameworks, for example, aheadgear, a headset, or a helmet.

In the eyeglasses example, eyewear device 100 includes a frame 105including a left rim 107A connected to a right rim 107B via a bridge 106adapted for a nose of the user. The left and right rims 107A-B includerespective apertures 175A-B, which hold a respective optical element180A-B, such as a lens and a display device. As used herein, the term“lens” is meant to cover transparent or translucent pieces of glass orplastic having curved and/or flat surfaces that cause light toconverge/diverge or that cause little or no convergence or divergence.

Although shown as having two optical elements 180A-B, the eyewear device100 can include other arrangements, such as a single optical element ormay not include any optical element 180A-B depending on the applicationor intended user of the eyewear device 100. As further shown, eyeweardevice 100 includes a left chunk 110A adjacent the left lateral side170A of the frame 105 and a right chunk 110B adjacent the right lateralside 170B of the frame 105. The chunks 110A-B may be integrated into theframe 105 on the respective sides 170A-B (as illustrated) or implementedas separate components attached to the frame 105 on the respective sides170A-B. Alternatively, the chunks 110A-B may be integrated into temples(not shown) attached to the frame 105.

In one example, the image display of optical assembly 180A-B includes anintegrated image display. As shown in FIG. 2B, the optical assembly180A-B includes a suitable display matrix 170 of any suitable type, suchas a liquid crystal display (LCD), an organic light-emitting diode(OLED) display, or any other such display. The optical assembly 180A-Balso includes an optical layer or layers 176, which can include lenses,optical coatings, prisms, mirrors, waveguides, optical strips, and otheroptical components in any combination. The optical layers 176A-N caninclude a prism having a suitable size and configuration and including afirst surface for receiving light from display matrix and a secondsurface for emitting light to the eye of the user. The prism of theoptical layers 176A-N extends over all or at least a portion of therespective apertures 175A-B formed in the left and right rims 107A-B topermit the user to see the second surface of the prism when the eye ofthe user is viewing through the corresponding left and right rims107A-B. The first surface of the prism of the optical layers 176A-Nfaces upwardly from the frame 105 and the display matrix overlies theprism so that photons and light emitted by the display matrix impingethe first surface. The prism is sized and shaped so that the light isrefracted within the prism and is directed towards the eye of the userby the second surface of the prism of the optical layers 176A-N. In thisregard, the second surface of the prism of the optical layers 176A-N canbe convex to direct the light towards the center of the eye. The prismcan optionally be sized and shaped to magnify the image projected by thedisplay matrix 170, and the light travels through the prism so that theimage viewed from the second surface is larger in one or more dimensionsthan the image emitted from the display matrix 170.

In another example, the image display device of optical assembly 180A-Bincludes a projection image display as shown in FIG. 2C. The opticalassembly 180A-B includes a laser projector 150, which is a three-colorlaser projector using a scanning mirror or galvanometer. Duringoperation, an optical source such as a laser projector 150 is disposedin or on one of the temples 125A-B of the eyewear device 100. Opticalassembly 180A-B includes one or more optical strips 155A-N spaced apartacross the width of the lens of the optical assembly 180A-B or across adepth of the lens between the front surface and the rear surface of thelens.

As the photons projected by the laser projector 150 travel across thelens of the optical assembly 180A-B, the photons encounter the opticalstrips 155A-N. When a particular photon encounters a particular opticalstrip, the photon is either redirected towards the user's eye, or itpasses to the next optical strip. A combination of modulation of laserprojector 150, and modulation of optical strips, may control specificphotons or beams of light. In an example, a processor controls opticalstrips 155A-N by initiating mechanical, acoustic, or electromagneticsignals. Although shown as having two optical assemblies 180A-B, theeyewear device 100 can include other arrangements, such as a single orthree optical assemblies, or the optical assembly 180A-B may havearranged different arrangement depending on the application or intendeduser of the eyewear device 100.

As further shown in FIGS. 2B-C, eyewear device 100 includes a left chunk110A adjacent the left lateral side 170A of the frame 105 and a rightchunk 110B adjacent the right lateral side 170B of the frame 105. Thechunks 110A-B may be integrated into the frame 105 on the respectivelateral sides 170A-B (as illustrated) or implemented as separatecomponents attached to the frame 105 on the respective sides 170A-B.Alternatively, the chunks 110A-B may be integrated into temples 125A-Battached to the frame 105. As used herein, the chunks 110A-B can includean enclosure that encloses a collection of processing units, camera,sensors, etc. (e.g., different for the right and left side) that areencompassed in an enclosure.

In one example, the image display includes a first (left) image displayand a second (right) image display. Eyewear device 100 includes firstand second apertures 175A-B, which hold a respective first and secondoptical assembly 180A-B. The first optical assembly 180A includes thefirst image display (e.g., a display matrix 170A of FIG. 2B; or opticalstrips 155A-N′ and a projector 150A of FIG. 2C). The second opticalassembly 180B includes the second image display e.g., a display matrix170B of FIG. 2B; or optical strips 155A-N″ and a projector 150B of FIG.2C).

FIG. 3 shows a rear perspective sectional view of the eyewear device ofFIG. 2A depicting an infrared camera 220, a frame front 330, a frameback 335, and a circuit board. It can be seen that the upper portion ofthe left rim 107A of the frame 105 of the eyewear device 100 includes aframe front 330 and a frame back 335. The frame front 330 includes afront-facing side configured to face outwards away from the eye of theuser. The frame back 335 includes a rear-facing side configured to faceinwards towards the eye of the user. An opening for the infrared camera220 is formed on the frame front 330.

As shown in the encircled cross-section 4-4 of the upper middle portionof the left rim 107A of the frame 105, a circuit board, which is aflexible printed circuit board (PCB) 340, is sandwiched between theframe front 330 and the frame back 335. Also shown in further detail isthe attachment of the left chunk 110A to the left temple 325A via a lefthinge 326A. In some examples, components of the depth sensor 213,including the infrared camera 220, the flexible PCB 340, or otherelectrical connectors or contacts may be located on the left temple 325Aor the left hinge 326A.

In an example, the left chunk 110A includes a chunk body 311, a chunkcap 312, an inward facing surface 391 and an outward facing surface 392(labeled, but not visible). Disposed inside the left chunk 110A arevarious interconnected circuit boards, such as PCBs or flexible PCBs,which include controller circuits for charging a battery, inwards facinglight emitting diodes (LEDs), and outwards (forward) facing LEDs.Although shown as being formed on the circuit boards of the left rim107A, the depth sensor 213, including the infrared emitter 215 and theinfrared camera 220, can be formed on the circuit boards of the rightrim 107B to captured infrared images utilized in the generation ofthree-dimensional depth images, for example, in combination with rightvisible light camera 114B.

FIG. 4 is a cross-sectional view through the infrared camera 220 and theframe corresponding to the encircled cross-section 4-4 of the eyeweardevice of FIG. 3. Various layers of the eyewear device 100 are visiblein the cross-section of FIG. 4. As shown, the flexible PCB 340 isdisposed on the frame back 335 and connected to the frame front 330. Theinfrared camera 220 is disposed on the flexible PCB 340 and covered byan infrared camera cover lens 445. For example, the infrared camera 220is reflowed to the back of the flexible PCB 340. Reflowing attaches theinfrared camera 220 to electrical contact pad(s) formed on the back ofthe flexible PCB 340 by subjecting the flexible PCB 340 to controlledheat, which melts a solder paste to connect the two components. In oneexample, reflowing is used to surface mount the infrared camera 220 onthe flexible PCB 340 and electrically connect the two components.However, it should be understood that through-holes can be used toconnect leads from the infrared camera 220 to the flexible PCB 340 viainterconnects, for example.

The frame front 330 includes an infrared camera opening 450 for theinfrared camera cover lens 445. The infrared camera opening 450 isformed on a front-facing side of the frame front 330 that is configuredto face outwards away from the eye of the user and towards a scene beingobserved by the user. In the example, the flexible PCB 340 can beconnected to the frame back 335 via a flexible PCB adhesive 460. Theinfrared camera cover lens 445 can be connected to the frame front 330via infrared camera cover lens adhesive 455. The connection can beindirect via intervening components.

FIG. 5 shows a rear perspective view of the eyewear device of FIG. 2A.The eyewear device 100 includes an infrared emitter 215, infrared camera220, a frame front 330, a frame back 335, and a circuit board 340. As inFIG. 3, it can be seen in FIG. 5 that the upper portion of the left rimof the frame of the eyewear device 100 includes the frame front 330 andthe frame back 335. An opening for the infrared emitter 215 is formed onthe frame front 330.

As shown in the encircled cross-section 6-6 in the upper middle portionof the left rim of the frame, a circuit board, which is a flexible PCB340, is sandwiched between the frame front 330 and the frame back 335.Also shown in further detail is the attachment of the left chunk 110A tothe left temple 325A via the left hinge 326A. In some examples,components of the depth sensor 213, including the infrared emitter 215,the flexible PCB 340, or other electrical connectors or contacts may belocated on the left temple 325A or the left hinge 326A.

FIG. 6 is a cross-sectional view through the infrared emitter 215 andthe frame corresponding to the encircled cross-section 6-6 of theeyewear device of FIG. 5. Multiple layers of the eyewear device 100 areillustrated in the cross-section of FIG. 6, as shown the frame 105includes the frame front 330 and the frame back 335. The flexible PCB340 is disposed on the frame back 335 and connected to the frame front330. The infrared emitter 215 is disposed on the flexible PCB 340 andcovered by an infrared emitter cover lens 645. For example, the infraredemitter 215 is reflowed to the back of the flexible PCB 340. Reflowingattaches the infrared emitter 215 to contact pad(s) formed on the backof the flexible PCB 340 by subjecting the flexible PCB 340 to controlledheat, which melts a solder paste to connect the two components. In oneexample, reflowing is used to surface mount the infrared emitter 215 onthe flexible PCB 340 and electrically connect the two components.However, it should be understood that through-holes can be used toconnect leads from the infrared emitter 215 to the flexible PCB 340 viainterconnects, for example.

The frame front 330 includes an infrared emitter opening 650 for theinfrared emitter cover lens 645. The infrared emitter opening 650 isformed on a front-facing side of the frame front 330 that is configuredto face outwards away from the eye of the user and towards a scene beingobserved by the user. In the example, the flexible PCB 340 can beconnected to the frame back 335 via the flexible PCB adhesive 460. Theinfrared emitter cover lens 645 can be connected to the frame front 330via infrared emitter cover lens adhesive 655. The coupling can also beindirect via intervening components.

FIG. 7 depicts an example of an emitted pattern of infrared light 781emitted by an infrared emitter 215 of the depth sensor 213. As shown,reflection variations of the emitted pattern of infrared light 782 arecaptured by the infrared camera 220 of the depth sensor 213 of theeyewear device 100 as an infrared image. The reflection variations ofthe emitted pattern of infrared light 782 is utilized to measure depthof pixels in a raw image (e.g., left raw image) to generate athree-dimensional depth image, such as the depth image.

Depth sensor 213 in the example includes the infrared emitter 215 toproject a pattern of infrared light and the infrared camera 220 tocapture infrared images of distortions of the projected infrared lightby objects or object features in a space, shown as scene 715 beingobserved by the wearer of the eyewear device 100. The infrared emitter215, for example, may blast infrared light 781, which falls on objects,or object features within the scene 715 like a sea of dots. In someexamples, the infrared light is emitted as a line pattern, a spiral, ora pattern of concentric rings or the like. Infrared light is typicallynot visible to the human eye. The infrared camera 220 is similar to astandard red, green, and blue (RGB) camera but receives and capturesimages of light in the infrared wavelength range. For depth sensing, theinfrared camera 220 is coupled to an image processor (element 912 ofFIG. 9) and the wallpaper programming (element 945) that judge time offlight based on the captured infrared image of the infrared light. Forexample, the distorted dot pattern 782 in the captured infrared imagecan then be processed by an image processor to determine depth from thedisplacement of dots. Typically, nearby objects or object features havea pattern with dots spread further apart and far away objects have adenser dot pattern. It should be understood that the foregoingfunctionality can be embodied in programming instructions of wallpaperprogramming or application (element 945) found in one or more componentsof the system.

FIG. 8A depicts an example of infrared light captured by the infraredcamera 220 of the depth sensor 213 with a left infrared camera field ofview 812. Infrared camera 220 captures reflection variations in theemitted pattern of infrared light 782 in the three-dimensional scene 715as an infrared image 859. As further shown, visible light is captured bythe left visible light camera 114A with a left visible light camerafield of view 111A as a left raw image 858A. Based on the infrared image859 and left raw image 858A, the three-dimensional depth image of thethree-dimensional scene 715 is generated.

FIG. 8B depicts an example of visible light captured by the left visiblelight camera 114A and visible light captured with a right visible lightcamera 114B. Visible light is captured by the left visible light camera114A with a left visible light camera field of view 111A as a left rawimage 858A. Visible light is captured by the right visible light camera114B with a right visible light camera field of view 111B as a right rawimage 858B. Based on the left raw image 858A and the right raw image858B, the three-dimensional depth image of the three-dimensional scene715 is generated.

FIG. 9 is a high-level functional block diagram of an example wallpapersystem 900, which includes a wearable device (e.g., the eyewear device100), a mobile device 990, and a server system 998 connected via variousnetworks. Eyewear device 100 includes a depth-capturing camera, such asat least one of the visible light cameras 114A-B; and the depth sensor213, shown as infrared emitter 215 and infrared camera 220. Thedepth-capturing camera can alternatively include at least two visiblelight cameras 114A-B (one associated with the left lateral side 170A andone associated with the right lateral side 170B). Depth-capturing cameragenerates depth images 962A-H, which are rendered three-dimensional (3D)models that are texture mapped images of a red, green, and blue (RGB)imaged scene, e.g., derived from the raw images 858A-N and processed(e.g., rectified) images 960A-N.

Mobile device 990 may be a smartphone, tablet, laptop computer, accesspoint, or any other such device capable of connecting with eyeweardevice 100 using both a low-power wireless connection 925 and ahigh-speed wireless connection 937. Mobile device 990 is connected toserver system 998 and network 995. The network 995 may include anycombination of wired and wireless connections.

Eyewear device 100 further includes two image displays of the opticalassembly 180A-B (one associated with the left lateral side 170A and oneassociated with the right lateral side 170B). Eyewear device 100 alsoincludes image display driver 942, image processor 912, low-powercircuitry 920, and high-speed circuitry 930. Image display of opticalassembly 180A-B are for presenting images and videos, including eightoriginal images 965A-H (e.g., raw images 858A-N and processed images960A-B) of the original video 964 and eight sets of wallpaper images968A-K, 969A-K, 975A-K of the wallpaper video 967 associated with arespective one of the eight original images 965A-H. Each of the sets ofwallpaper images 968A-K, 969A-K, 975A-K has eleven wallpaper images(e.g., 968A-K) to provide a total of eleven fields of view asrepresented by the spatial movement parameter 976A-K (e.g., 0.0, 0.1, .. . 1.0) of the respective original image 965A-H. The middle wallpaperimage (e.g., 968F) may correspond to the field of view of the originalimage (e.g., 965A). Image display driver 942 is coupled to the imagedisplay of optical assembly 180A-B to control the image display ofoptical assembly 180A-B to present the images and videos, such asselective images of the original video 964 and the wallpaper video 967.Eyewear device 100 further includes a user input device 991 (e.g., touchsensor) to receive a spatial user input selection 978 and a time userinput selection 979 from a user. In some examples, the user input device991 includes a movement tracker 981 (e.g., an inertial measurementunit).

The components shown in FIG. 9 for the eyewear device 100 are located onone or more circuit boards, for example a PCB or flexible PCB, in therims or temples. Alternatively, or additionally, the depicted componentscan be located in the chunks, frames, hinges, or bridge of the eyeweardevice 100. Left and right visible light cameras 114A-B can includedigital camera elements such as a complementarymetal-oxide-semiconductor (CMOS) image sensor, charge coupled device, alens, or any other respective visible or light capturing elements thatmay be used to capture data, including images of scenes with unknownobjects.

Eyewear device includes 100 includes a memory 934 which includeswallpaper programming 945 to perform a subset or all of the functionsdescribed herein for wallpaper effects, in which one or both of: (i) aspatial user input selection 978, and (ii) a time user input selection979 from a user are applied to interact with wallpaper images 968A-K,969A-K, . . . 975A-K. As shown, memory 934 further includes a left rawimage 858A captured by left visible light camera 114A, a right raw image858B captured by right visible light camera 114B, and an infrared image859 captured by infrared camera 220 of the depth sensor 213. Memory 934further includes multiple depth images 962A-H, one for each of the eightoriginal images 965A-H. Depth images 962A-H are generated, via thedepth-capturing camera, and each of the depth images 962A-H includes arespective mesh of vertices 963A-H.

A flowchart outlining functions which can be implemented in thewallpaper programming 945 is shown in FIG. 11. Memory 934 furtherincludes the spatial user input selection 978 (e.g., tilt of the eyeweardevice 100 or the mobile device 990) and the time user input selection979 (e.g., an initial touch point and a final touch point), which arereceived by the user input device 991. Memory 934 further includes: aleft image disparity map 961A, a right image disparity map 961B, a leftprocessed (e.g., rectified) image 960A and a right processed (e.g.,rectified) image 960B (e.g., to remove vignetting towards the end of thelens). As further shown, memory 934 includes the respective mesh ofvertices 963A-H for each of the depth images 962A-H; and an originalvideo 964 that includes a sequence of original images 965A-H andrespective time coordinates 966A-H associated with each of the originalimages 965A-H. Memory further includes a wallpaper video 967 thatincludes eight sets of eleven wallpaper images 968A-K, 969A-K, . . .975A-K for each of the eight respective original images 965A-H. Arespective set of the eleven wallpaper images 968A-K, 969A-K, 975A-K isassociated with the respective time coordinate 966A-H of the respectiveoriginal image 965A-H. Some or all of the stored information in thememory 934 can be generated during image processing of the raw images858A-B to generate wallpaper images 968A-K, 969A-K, . . . 975A-K.

For example, K fields of view (or viewpoints), 0.1, 0.2, until 1 arecreated to generate eleven views corresponding to eleven wallpaperimages 968A-K, such that each wallpaper image 968A-K is seen at adifferent angular orientation. When the user tilts (e.g., rotates) themobile device 990, the eleven different wallpaper images 968A-K arerotated between. By having several (eleven) wallpaper images 968A-K, a3D appearance is created. Moving the mobile device 990 provides aneffect of different viewpoints, which provides a short video animationeven when based on just a single original image 965A set as wallpaper.

In the generation of the sets of wallpaper images 968A-K, 969A-K, . . .975A-K based on 3D image processing, actual distance or depth can beused to rotate around the 3D mesh of vertices 963A-H. In the 2D lightfield effect image processing version, disparity is used, which isrelated to depth, but disparity is not directly depth. Rather, disparityis just a movement of pixels, which means the image processing can bedone in the 2D space to speed up runtime and reduce memory requirements.There need not be any transformation into 3D, rather there arecorresponding pixels and interpolation between the corresponding pixels.While the correspondence (disparity), can translate into depth(distance), depth is not needed for this spatial movement wallpapereffect. Whether the depth on the Z axis is 10 meters or 20 meters doesnot matter, the pixel can be moved to a different X axis locationcoordinate depending on the spatial movement input selection 978.

As shown in FIG. 9, high-speed circuitry 930 includes high-speedprocessor 932, memory 934, and high-speed wireless circuitry 936. In theexample, the image display driver 942 is coupled to the high-speedcircuitry 930 and operated by the high-speed processor 932 in order todrive the left and right image displays of the optical assembly 180A-B.High-speed processor 932 may be any processor capable of managinghigh-speed communications and operation of any general computing systemneeded for eyewear device 100. High-speed processor 932 includesprocessing resources needed for managing high-speed data transfers onhigh-speed wireless connection 937 to a wireless local area network(WLAN) using high-speed wireless circuitry 936. In certain embodiments,the high-speed processor 932 executes an operating system such as aLINUX operating system or other such operating system of the eyeweardevice 100 and the operating system is stored in memory 934 forexecution. In addition to any other responsibilities, the high-speedprocessor 932 executing a software architecture for the eyewear device100 is used to manage data transfers with high-speed wireless circuitry936. In certain embodiments, high-speed wireless circuitry 936 isconfigured to implement Institute of Electrical and Electronic Engineers(IEEE) 802.11 communication standards, also referred to herein as Wi-Fi.In other embodiments, other high-speed communications standards may beimplemented by high-speed wireless circuitry 936.

Low-power wireless circuitry 924 and the high-speed wireless circuitry936 of the eyewear device 100 can include short range transceivers(Bluetooth™) and wireless wide, local, or wide area network transceivers(e.g., cellular or WiFi). Mobile device 990, including the transceiverscommunicating via the low-power wireless connection 925 and high-speedwireless connection 937, may be implemented using details of thearchitecture of the eyewear device 100, as can other elements of network995.

Memory 934 includes any storage device capable of storing various dataand applications, including, among other things, camera data generatedby the left and right visible light cameras 114A-B, infrared camera 220,and the image processor 912, as well as images generated for display bythe image display driver 942 on the image displays of the opticalassembly 180A-B. While memory 934 is shown as integrated with high-speedcircuitry 930, in other embodiments, memory 934 may be an independentstandalone element of the eyewear device 100. In certain suchembodiments, electrical routing lines may provide a connection through achip that includes the high-speed processor 932 from the image processor912 or low-power processor 922 to the memory 934. In other embodiments,the high-speed processor 932 may manage addressing of memory 934 suchthat the low-power processor 922 will boot the high-speed processor 932any time that a read or write operation involving memory 934 is needed.

As shown in FIG. 9, the processor 932 of the eyewear device 100 can becoupled to the depth-capturing camera (visible light cameras 114A-B; orvisible light camera 114A, infrared emitter 215, and infrared camera220), the image display driver 942, the user input device 991, and thememory 934. As shown in FIG. 10, the processor 1030 of the mobile device990 can be coupled to the depth-capturing camera 1070, the image displaydriver 1090, the user input device 1091, and the memory 1040A. Eyeweardevice 100 can perform all or a subset of any of the following functionsdescribed below as a result of the execution of the wallpaperprogramming 945 in the memory 934 by the processor 932 of the eyeweardevice 100. Mobile device 990 can perform all or a subset of any of thefollowing functions described below as a result of the execution of thewallpaper programming 945 in the memory 1040A by the processor 1030 ofthe mobile device 990. Functions can be divided in the wallpaper system900, such that the eyewear device 100 generates the raw images 858A-B,but the mobile device 990 performs the remainder of the image processingon the raw images 858A-B to generate the original video, including theeight original images 965A-H, and the wallpaper video 967, including theeight sets of wallpaper images 968A-K, 969A-K, . . . 975A-K.

An example wallpaper system 900 includes an image display 180A-B, 1080for presenting a wallpaper video 967 including a sequence of wallpaperimages 968A-K, 969A-K, . . . 975A-K. The wallpaper images 968A-K,969A-K, . . . 975A-K are two-dimensional (2D) and based on raw images858A-B or processed raw images 960A-B captured via the depth-capturingcamera. Each of the wallpaper images 968A-K, 969A-K, . . . 975A-K isassociated with a respective time coordinate 966A-H on a time (T) axisfor a presentation time and a respective spatial movement parameter976A-K. Wallpaper system 900 further includes an image display driver942, 1090 coupled to the image display 180A-B, 1080 to control the imagedisplay 180A-B, 1080 to present the wallpaper video 967.

Wallpaper system 900 further includes a user input device 991, 1091 toreceive from a user one or both of: (i) a spatial user input selection978, and (ii) a time user input selection 979 to apply to the wallpapervideo 967. Wallpaper system 900 further includes a memory 934, 1040A;and a processor 932, 1030 coupled to the image display driver 942, 1090the user input device 991, 1091, and the memory 934, 1040A. Wallpapersystem 900 further includes wallpaper programming 945, 1045 in thememory 934, 1040A.

Execution of the wallpaper programming 945 by the processor 932, 1030configures the wallpaper system 900 to perform functions, includingfunctions to present, via the image display 180A-B, 1080, a firstwallpaper image of the wallpaper video 967 to the user. Let's assume inthe example that the first wallpaper image is 968A that is associatedwith the spatial movement parameter 976A. Wallpaper system 900 receives,via the user input device 991, 1091 from the user one or both of: (i)the spatial user input selection 978, and (ii) the time user inputselection 979 to apply to the wallpaper video 967.

Wallpaper system 900 determines one or both of: (i) the respectivespatial movement parameter 976A-K associated with the spatial user inputselection 978, and (ii) the respective time coordinate 966A-H associatedwith the time user input selection 979. Wallpaper system 900 presents,via the image display 180A-B, 1080, a second wallpaper image associatedwith one or both of: (i) the respective spatial movement parameter976A-K, and (ii) the respective time coordinate 966A-H. Continuing withthe example where the first wallpaper image is 968A, assume that thespatial movement parameter is 976B and the time coordinate is 966B. Ifthere is only the spatial user input selection 978 for just spatialmovement, then the second wallpaper image is 968B to just presentspatial movement of the wallpaper video 967. If there is only the timeuser input selection 979 for just time movement, then the secondwallpaper image is 969A to present just time movement of the wallpapervideo 967. However, if there is both the spatial user input selection978 and the time user input selection 979, the second wallpaper image is969B to present both spatial movement and time movement of the wallpapervideo 967.

Either the mobile device 990 or eyewear device 100 can include the userinput device 991, 1091. In the touch based user input device 991, 1091,the time user input selection 979 may be detected as vertical scrolling(swiping) type of finger gesture on the touch sensor (e.g., upwardsscrolling to advance (forward) the time coordinate 966A-H or downwardsscrolling to rewind the time coordinate 966A-H. A touch-based user inputdevice 1091 can be integrated into the mobile device 990 as a touchscreen display. In one example, the user input device 991, 1091 includesa touch sensor including an input surface and a sensor array that iscoupled to the input surface to receive at least one finger contactinputted from a user. User input device 991, 1091 further includes asensing circuit integrated into or connected to the touch sensor andconnected to the processor 932, 1030. The sensing circuit is configuredto measure voltage to track the at least one finger contact on the inputsurface. The function of receiving, via the user input device 1091 fromthe user one or both of: (i) the spatial user input selection 978, and(ii) the time user input selection 979 to apply to the wallpaper video967 includes the following functions. First, receiving on the inputsurface of the touch sensor the at least one finger contact inputtedfrom the user. Second, tracking, via the sensing circuit, the at leastone finger contact on the input surface. Third, detecting one or bothof: (i) the spatial user input selection 978, and (ii) the time userinput selection 979 to apply to the wallpaper video 967 on the inputsurface of the touch sensor based on the at least one finger contactfrom the user.

A touch-based user input device 991 can be integrated into the eyeweardevice 100. As noted above, eyewear device 100 includes a chunk 110A-Bintegrated into or connected to the frame 105 on the lateral side 170A-Bof the eyewear device 100. The frame 105, the temple 125A-B, or thechunk 110A-B includes a circuit board that includes the touch sensor.The circuit board includes a flexible printed circuit board. The touchsensor is disposed on the flexible printed circuit board. The sensorarray is a capacitive array or a resistive array. The capacitive arrayor the resistive array includes a grid that forms a two-dimensionalrectangular coordinate system to track X and Y axes locationcoordinates.

User input device 991, 1091 of the mobile device 990 or eyewear device100 can include the movement tracker 981. In the movement tracker 981 ofthe user input device 991, 1091, the spatial user input selection 978may be detected as left or right horizontal titling (change of tiltangle or rotation angle), for example, of the mobile device 990 oreyewear device 100. Movement (movt) tracker 981 is an electronic device,such as an inertial measurement unit (IMU), that measures and reports abody's specific force, angular rate, and sometimes the magnetic fieldsurrounding the body, using a combination of accelerometers andgyroscopes, sometimes also magnetometers. If a magnetometer is present,the magnetic field can be used as input to detect specific gestures thatare dependent on Earth's or an artificial magnetic field. In thisexample, the inertial measurement unit determines a rotationacceleration of the eyewear device 100 or a host computer, such as themobile device 990. The movement tracker 981 works by detecting linearacceleration using one or more accelerometers and rotational rate usingone or more gyroscopes. Typical configurations of inertial measurementunits contain one accelerometer, gyroscope, and magnetometer per axisfor each of the three axes: horizontal axis for left-right movement (X),vertical axis (Y) for top-bottom movement, and depth or distance axisfor up-down movement (Z). The gyroscope detects the rate of rotationaround 3 axes (X, Y, and Z). The magnetometer detects the magnetic field(e.g., facing south, north, etc.) like a compass which generates aheading reference, which is a mixture of Earth's magnetic field andother artificial magnetic field (such as ones generated by power lines).The three accelerometers detect acceleration along the horizontal (X),vertical (Y), and depth or distance (Z) axes defined above, which can bedefined relative to the ground, the eyewear device 100 or the mobiledevice 990, the depth-capturing camera, or the user wearing the eyeweardevice 100 or holding (or carrying) the mobile device 990. Thus, theaccelerometer detects a 3-axis acceleration vector, which then can beused to detect Earth's gravity vector.

In one example, the mobile device 990 includes the user input device1091. The user input device 1091 includes the movement tracker 981coupled to the processor 1030 to track movement of the mobile device990. Movement tracker 981 includes: (i) at least one accelerometer tomeasure acceleration of the mobile device, (ii) at least one gyroscopeto measure rotation of the mobile device, or (iii) an inertialmeasurement unit (IMU) having the at least one accelerometer and the atleast one gyroscope. The function of receiving, via the user inputdevice 1091 from the user one or both of: (i) the spatial user inputselection 978, and (ii) the time user input selection 979 from the userto apply to the wallpaper video 967 includes the following functions.First, tracking, via the movement tracker 981, movement of the mobiledevice 990 by: (i) measuring, via the at least one accelerometer, theacceleration of the mobile device 990, (ii) measuring, via the at leastone gyroscope, the rotation of the mobile device 990, or (iii)measuring, via the inertial measurement unit, both the acceleration andthe rotation of the mobile device 990. Second, detecting one or both of:(i) the spatial user input selection 978, and (ii) the time user inputselection 979 to apply to the wallpaper video 967 by detecting at leastone variation of the tracked movement over a time period.

As noted above, eyewear device 100 includes a frame 105, a temple 125A-Bconnected to a lateral side 170A-B of the frame 105, and thedepth-capturing camera. The depth-capturing camera is supported by atleast one of the frame 105 or the temple 125A-B. The depth-capturingcamera includes: (i) at least two visible light cameras 114A-B withoverlapping fields of view 111A-B, or (ii) a least one visible lightcamera 114A or 114B and a depth sensor 213. The depth-capturing camera1070 of the mobile device 990 can be similarly structured.

In one example, the depth-capturing camera includes the at least twovisible light cameras 114A-B comprised of a left visible light camera114A with a left field of view 111A to capture a left raw image 858A anda right visible light camera 114B with a right field of view 111B tocapture a right raw image 858B. The left field of view 111A and theright field of view 111B have an overlapping field of view 813 (see FIG.8B).

In a 2D image processing light field effects example, execution of thewallpaper programming 945 by the processor 932, 1030 configures thewallpaper system 900 to perform functions, including functions togenerate, the second wallpaper image by implementing the followingfunctions. First, calculating: (i) a left image disparity map 961Abetween a left pixel matrix of pixels and a right pixel matrix ofpixels, and (ii) a right image disparity map 961B between the rightpixel matrix and the left pixel matrix. The left raw image 858A or aleft processed image 960A include the left pixel matrix. The right rawimage 858B or a right processed image 960B include the right pixelmatrix. Second, determining the respective spatial movement parameter976A-K of the left pixel matrix and the right pixel matrix along atleast one of: (i) an X axis for horizontal position movement, and (ii) aY axis for vertical position movement. In one example, the movement ismade along only one single X axis that is parallel to the baseline (animaginary line connecting the visible light cameras 114A-B). Third,filling up a left interpolated pixel matrix by moving pixels in the leftpixel matrix along at least one of: (i) the X axis, and (ii) the Y axisbased on the respective spatial movement parameter 976A-K. Fourth,filling up a right interpolated pixel matrix by moving pixels in theright pixel matrix along at least one of: (i) the X axis, and (ii) the Yaxis based on the spatial movement parameter 976A-K. Fifth, blendingtogether the left interpolated pixel matrix and the right interpolatedpixel matrix to create the second wallpaper image.

In the 2D image processing example, assume that the first wallpaperimage is 968F, which is associated with the spatial movement parameterof 976F. The first wallpaper image 968F has the same field of view ofthe original image 965A and essentially mimics the original image 965A.Thus, the raw images 858A-B as captured by the left and right visiblelight cameras 114A-B correspond to the original image 965A beforeprocessing and the processed images 960A-B correspond to the originalimage 965A after rectification. Assuming that the received spatial userinput selection 978 is for just spatial movement and is associated withthe spatial movement parameter is 976B, then the presented secondwallpaper image is 968B. The second wallpaper image 968B is blendedtogether based on the left interpolated pixel matrix and the rightinterpolated pixel matrix to present an appearance of spatial movementor rotation around the first wallpaper image 968F (i.e., original image965A). Blending together the left interpolated pixel matrix and theright interpolated pixel matrix is based on disparity confidence levels,gradients, or combination thereof in the left image disparity map 961Aand the right image disparity map 961B. The disparity confidence levelvalue is based, for instance, on the magnitude of correlation betweenthe left and the right pixels.

Continuing with the 2D example, the spatial movement parameter 976A-K isfor horizontal position movement along the X axis. The left interpolatedpixel matrix is filled up by moving pixels in the left pixel matrixalong the X axis. The right interpolated pixel matrix is filled up bymoving pixels in the right pixel matrix along the X axis further basedon a complement of the spatial movement parameter 976A-K (i.e., 1 minusthe spatial movement parameter 976A-K).

In some examples, it may be advantageous to pre-process the originalimages 965A-H to create a wallpaper image matrix. Creating the wallpaperimage matrix improves processing time of the wallpaper system 900 torespond to the spatial user input selection 978 and the time user inputselection 979 by pre-generating and persistently storing as an archiveeight sets of wallpaper images 968A-K, 969A-K, . . . 975A-K. On theother hand, pre-generating the wallpaper image matrix imposes additionalmemory 932, 1040A storage requirements on the wallpaper system 900, butdoes provide the benefit of faster response time to user interactions tomanipulate the wallpaper video 967. Hence, execution of the wallpaperprogramming 945 by the processor 932, 1030 configures the wallpapersystem 900 to upon selection of the original video 964 as wallpaper bythe user, create and persistently store in the memory 934, 1040A awallpaper image matrix. Each respective set of wallpaper images 968A-K,969A-K, . . . 975A-K corresponds to the respective time coordinate966A-H of a respective original image 965A-H of the original video 964.Each respective set of wallpaper images 968A-K, 969A-K, . . . 975A-Kprovides an appearance of a spatial movement or rotation around therespective original image 965A-H of the original video 964. Eachwallpaper image 968A-K within the respective set of wallpaper images968A-K, 969A-K, . . . 975A-K corresponds to a different spatial movementparameter 976A-K (e.g., there are eleven fields of view) within therespective original image 965A-H.

In the 2D image processing light field effects example, the function ofcreating, the wallpaper image matrix, including the sets of wallpaperimages 968A-K, 969A-K, . . . 975A-K of the wallpaper video 967 includesthe following functions. First, capturing, via the depth-capturingcamera, the left raw image 858A and the right raw image 858Bcorresponding to the respective original image 965A-H. Second,calculating: (i) a left image disparity map 961A between a left pixelmatrix of pixels and a right pixel matrix of pixels, and (ii) a rightimage disparity map 961B between the right pixel matrix and the leftpixel matrix. The left raw image 858A or a left processed image 960Ainclude the left pixel matrix. The right raw image 858B or a rightprocessed image 960B include the right pixel matrix. Then for eachwallpaper image 968A-K, 969A-K, . . . 975A-K of the respective set ofwallpaper images 968A-K, 969A-K, . . . 975A-K implement the followingfunctions. First, determining the respective spatial movement parameter976A-K of the left pixel matrix and the right pixel matrix of therespective original image 965A-H along at least one of: (i) an X axisfor horizontal position movement, and (ii) a Y axis for verticalposition movement. Second, filling up a left interpolated pixel matrixby moving pixels in the left pixel matrix along at least one of: (i) theX axis, and (ii) the Y axis based on the respective spatial movementparameter 976A-K for the respective original image 965A-H. Third,filling up a right interpolated pixel matrix by moving pixels in theright pixel matrix along at least one of: (i) the X axis, and (ii) the Yaxis based on the respective spatial movement parameter 976A-K of therespective original image 965A-H. Fourth, blending together the leftinterpolated pixel matrix and the right interpolated pixel matrix tocreate the respective wallpaper image 968A-K, 969A-K, . . . 975A-K.

In the 3D image processing example, execution of the wallpaperprogramming 945 by the processor 932, 1030 configures the wallpapersystem 900 to perform the following functions. First, create, via thedepth-capturing camera, a respective depth image 962A-H corresponding tothe first wallpaper image. The respective depth image 962A-H is formedof a respective mesh of vertices 963A-H. Each vertex represents a pixelin a three-dimensional scene. Each vertex has a position attribute. Theposition attribute of each vertex is based on a three-dimensionallocation coordinate system and includes an X location coordinate on an Xaxis for horizontal position, a Y location coordinate on a Y axis forvertical position, and a Z location coordinate on a Z axis for depth(distance). Each vertex further includes one or more of a colorattribute, a texture attribute, or a reflectance attribute. The functionof generating, the presented second wallpaper image is implemented byrotating the respective depth image 962A-H based on the respectivespatial movement parameter 976A-K. The respective spatial movementparameter 976A-K is along at least one of: (i) the X axis for horizontalposition movement, (ii) the Y axis for vertical position movement, and(iii) the Z axis for depth (distance) movement.

In one example of the wallpaper system 900, the processor comprises afirst processor 932 and a second processor 1030. The memory comprises afirst memory 934 and a second memory 1040A. The eyewear device 100includes a first network communication 924 or 936 interface forcommunication over a network 925 or 937 (e.g., a wireless short-rangenetwork or a wireless local area network), the first processor 932coupled to the first network communication interface 924 or 936, and thefirst memory 934 accessible to the first processor 932. Eyewear device100 further includes wallpaper programming 945 in the first memory 934.Execution of first wallpaper programming 945 by the first processor 932configures the eyewear device 100 to perform functions to capture, viathe depth-capturing camera, the original images 965A-H (e.g., raw images858A-B or the processed raw images 960A-B).

The wallpaper system 900 further comprises a host computer, such as themobile device 990, coupled to the eyewear device 100 over the network925 or 937. The host computer includes a second network communicationinterface 1010 or 1020 for communication over the network 925 or 937.The second processor 1030 is coupled to the second network communicationinterface 1010 or 1020. The second memory 1040A is accessible to thesecond processor 1030. Host computer further includes second wallpaperprogramming 945 in the second memory 1040A.

Execution of the second wallpaper programming 945 by the secondprocessor 1030 configures the host computer 990, 998 to perform thefollowing functions. First, present, via the image display 1080, thefirst wallpaper image of the wallpaper video 967 to the user. Second,receive, via the user input device 1091, 981 (e.g., touch screen, acomputer mouse, movement tracker) from the user one or both of: (i) thespatial user input selection 978, and (ii) the time user input selection979 to apply to the wallpaper video 967. Third, in response to receivingone or both of: (i) the spatial user input selection 978, and (ii) thetime user input selection 979: determine one or both of: (i) therespective spatial movement parameter 976A-K associated with the spatialuser input selection 978, and (ii) the respective time coordinate 966A-Hassociated with the time user input selection 979. Fifth, present, viathe image display 180A-B, 1080, the second wallpaper image associatedwith one or both of: (i) the respective spatial movement parameter976A-K, and (ii) the respective time coordinate 966A-H.

Server system 998 may be one or more computing devices as part of aservice or network computing system, for example, that include aprocessor, a memory, and network communication interface to communicateover the network 995 with the mobile device 990 and eyewear device 100.Eyewear device 100 is connected with a host computer. For example, theeyewear device 100 is paired with the mobile device 990 via thehigh-speed wireless connection 937 or connected to the server system 998via the network 995.

Output components of the eyewear device 100 include visual components,such as the left and right image displays of optical assembly 180A-B asdescribed in FIGS. 2B-C (e.g., a display such as a liquid crystaldisplay (LCD), a plasma display panel (PDP), a light emitting diode(LED) display, a projector, or a waveguide). The image displays of theoptical assembly 180A-B are driven by the image display driver 942. Theoutput components of the eyewear device 100 further include acousticcomponents (e.g., speakers), haptic components (e.g., a vibratorymotor), other signal generators, and so forth. The input components ofthe eyewear device 100, the mobile device 990, and server system 998,may include alphanumeric input components (e.g., a keyboard, a touchscreen configured to receive alphanumeric input, a photo-opticalkeyboard, or other alphanumeric input components), point-based inputcomponents (e.g., a mouse, a touchpad, a trackball, a joystick, a motionsensor, or other pointing instruments), tactile input components (e.g.,a physical button, a touch screen that provides location and force oftouches or touch gestures, or other tactile input components), audioinput components (e.g., a microphone), and the like.

Eyewear device 100 may optionally include additional peripheral deviceelements. Such peripheral device elements may include biometric sensors,additional sensors, or display elements integrated with eyewear device100. For example, peripheral device elements may include any I/Ocomponents including output components, motion components, positioncomponents, or any other such elements described herein.

For example, the biometric components include components to detectexpressions (e.g., hand expressions, facial expressions, vocalexpressions, body gestures, or eye tracking), measure biosignals (e.g.,blood pressure, heart rate, body temperature, perspiration, or brainwaves), identify a person (e.g., voice identification, retinalidentification, facial identification, fingerprint identification, orelectroencephalogram based identification), and the like. The motioncomponents include acceleration sensor components (e.g., accelerometer),gravitation sensor components, rotation sensor components (e.g.,gyroscope), and so forth. The position components include locationsensor components to generate location coordinates (e.g., a GlobalPositioning System (GPS) receiver component), WiFi or Bluetooth™transceivers to generate positioning system coordinates, altitude sensorcomponents (e.g., altimeters or barometers that detect air pressure fromwhich altitude may be derived), orientation sensor components (e.g.,magnetometers), and the like. Such positioning system coordinates canalso be received over wireless connections 925 and 937 from the mobiledevice 990 via the low-power wireless circuitry 924 or high-speedwireless circuitry 936.

FIG. 10 is a high-level functional block diagram of an example of amobile device 990 that communicates via the wallpaper system 900 of FIG.9. Mobile device 990 includes a user input device 1091 (e.g., a touchscreen display or a movement tracker 981) to receive a spatial userinput selection 978 or a time user input selection 979 to apply to anoriginal image 965A-H in real-time, or a wallpaper video 967 that cancomprise an archived wallpaper image matrix. Mobile device 990responsively presents a respective wallpaper image 968A-K, 969A-K, . . .975A-K based on a respective spatial movement parameter 976A-Kassociated with the spatial user input selection 978 or a respectivetime coordinate 966A-H associated with the time user input selection979.

Mobile device 990 includes a flash memory 1040A which includes wallpaperprogramming 945 to perform all or a subset of the functions describedherein for producing wallpaper effects, in which one or both of: (i) thespatial user input selection 978, and (ii) the time user input selection979 from a user are applied to interact with wallpaper images 968A-K,969A-K, . . . 975A-K.

As shown, memory 1040A further includes a left raw image 858A capturedby left visible light camera 114A, a right raw image 858B captured byright visible light camera 114B, and an infrared image 859 captured byinfrared camera 220 of the depth sensor 213. Mobile device 1090 caninclude a depth-capturing camera 1070 that comprises at least twovisible light cameras (first and second visible light cameras withoverlapping fields of view) or at least on visible light camera and adepth sensor with substantially overlapping fields of view like theeyewear device 100. When the mobile device 990 includes components likethe eyewear device 100, such as the depth-capturing camera, the left rawimage 858A, the right raw image 858B, and the infrared image 859 can becaptured via the depth-capturing camera 1070 of the mobile device 990.

Memory 1040A further includes multiple depth images 962A-H (includingrespective meshes of vertices 963A-H), which are generated, via thedepth-capturing camera of the eyewear device 100 or via thedepth-capturing camera 1070 of the mobile device 990 itself. A flowchartoutlining functions which can be implemented in the wallpaperprogramming 945 is shown in FIG. 11. Memory 1040A further includes thespatial user input selection 978 and the time user input selection 979(e.g., such as an initial touch point and a final touch point) receivedby the user input device 1091. Memory 1040A further includes: a leftimage disparity map 961A, a right image disparity map 961B, and leftprocessed (e.g., rectified) and right processed (e.g., rectified) images960A-B (e.g., to remove vignetting towards the end of the lens). Asfurther shown, memory 1040A includes the original video 964 withoriginal images 965A-H and an associated respective time coordinate966A-H. Memory 1040A further includes wallpaper video 967 with sets ofwallpaper images 968A-K, 969A-K, . . . 975A-K and associated respectivespatial movement parameters 976A-K. Some or all of the storedinformation in the memory 1040A can be generated during image processingof the raw images 858A-B to generate the wallpaper images 968A-K,969A-K, . . . 975A-K.

As shown, the mobile device 990 includes an image display 1080, an imagedisplay driver 1090 to control the image display, and a user inputdevice 1091 similar to the eyewear device 100. In the example of FIG.10, the image display 1080 and user input device 1091 are integratedtogether into a touch screen display.

Examples of touch screen type mobile devices that may be used include(but are not limited to) a smart phone, a personal digital assistant(PDA), a tablet computer, a laptop computer, or other portable device.However, the structure and operation of the touch screen type devices isprovided by way of example; and the subject technology as describedherein is not intended to be limited thereto. For purposes of thisdiscussion, FIG. 10 therefore provides block diagram illustrations ofthe example mobile device 990 having a touch screen display fordisplaying content and receiving user input as (or as part of) the userinterface.

The activities that are the focus of discussions here typically involvedata communications related to receiving one or both of: (i) a spatialuser input selection 978, and (ii) a time user input selection 979 froma user to present a high degree of user interaction with an originalimage 965A-H or an original video 964 selected as wallpaper in theportable eyewear device 100 or the mobile device 990. As shown in FIG.10, the mobile device 990 includes at least one digital transceiver(XCVR) 1010, shown as WWAN XCVRs, for digital wireless communicationsvia a wide area wireless mobile communication network. The mobile device990 also includes additional digital or analog transceivers, such asshort range XCVRs 1020 for short-range network communication, such asvia NFC, VLC, DECT, ZigBee, Bluetooth™, or WiFi. For example, shortrange XCVRs 1020 may take the form of any available two-way wirelesslocal area network (WLAN) transceiver of a type that is compatible withone or more standard protocols of communication implemented in wirelesslocal area networks, such as one of the Wi-Fi standards under IEEE802.11 and WiMAX.

To generate location coordinates for positioning of the mobile device990, the mobile device 990 can include a global positioning system (GPS)receiver. Alternatively, or additionally the mobile device 990 canutilize either or both the short range XCVRs 1020 and WWAN XCVRs 1010for generating location coordinates for positioning. For example,cellular network, WiFi, or Bluetooth™ based positioning systems cangenerate very accurate location coordinates, particularly when used incombination. Such location coordinates can be transmitted to the eyeweardevice over one or more network connections via XCVRs 1010, 1020.

The transceivers 1010, 1020 (network communication interfaced) conformto one or more of the various digital wireless communication standardsutilized by modern mobile networks. Examples of WWAN transceivers 1010include (but are not limited to) transceivers configured to operate inaccordance with Code Division Multiple Access (CDMA) and 3rd GenerationPartnership Project (3GPP) network technologies including, for exampleand without limitation, 3GPP type 2 (or 3GPP2) and LTE, at timesreferred to as “4G.” For example, the transceivers 1010, 1020 providetwo-way wireless communication of information including digitized audiosignals, still image and video signals, web page information for displayas well as web related inputs, and various types of mobile messagecommunications to/from the mobile device 990 for wallpaper effects.

Several of these types of communications through the transceivers 1010,1020 and a network, as discussed previously, relate to protocols andprocedures in support of communications with the eyewear device 100 orthe server system 998 for generating original images 965A-H and sets ofwallpaper images 968A-K, 969A-K, . . . 975A-K, such as transmitting leftraw image 858A, right raw image 858B, infrared image 859, depth images962A-H, and processed (e.g., rectified) images 960A-B. Suchcommunications, for example, may transport packet data via the shortrange XCVRs 1020 over the wireless connections 925 and 937 to and fromthe eyewear device 100 as shown in FIG. 9. Such communications, forexample, may also transport data utilizing IP packet data transport viathe WWAN XCVRs 1010 over the network (e.g., Internet) 995 shown in FIG.9. Both WWAN XCVRs 1010 and short range XCVRs 1020 connect through radiofrequency (RF) send-and-receive amplifiers (not shown) to an associatedantenna (not shown).

The mobile device 990 further includes a microprocessor, shown as CPU1030, sometimes referred to herein as the host controller. A processoris a circuit having elements structured and arranged to perform one ormore processing functions, typically various data processing functions.Although discrete logic components could be used, the examples utilizecomponents forming a programmable CPU. A microprocessor for exampleincludes one or more integrated circuit (IC) chips incorporating theelectronic elements to perform the functions of the CPU. The processor1030, for example, may be based on any known or available microprocessorarchitecture, such as a Reduced Instruction Set Computing (RISC) usingan ARM architecture, as commonly used today in mobile devices and otherportable electronic devices. Of course, other processor circuitry may beused to form the CPU 1030 or processor hardware in smartphone, laptopcomputer, and tablet.

The microprocessor 1030 serves as a programmable host controller for themobile device 990 by configuring the mobile device 990 to performvarious operations, for example, in accordance with instructions orprogramming executable by processor 1030. For example, such operationsmay include various general operations of the mobile device, as well asoperations related to the wallpaper programming 945 and communicationswith the eyewear device 100 and server system 998. Although a processormay be configured by use of hardwired logic, typical processors inmobile devices are general processing circuits configured by executionof programming.

The mobile device 990 includes a memory or storage device system, forstoring data and programming. In the example, the memory system mayinclude a flash memory 1040A and a random access memory (RAM) 1040B. TheRAM 1040B serves as short term storage for instructions and data beinghandled by the processor 1030, e.g. as a working data processing memory.The flash memory 1040A typically provides longer term storage.

Hence, in the example of mobile device 990, the flash memory 1040A isused to store programming or instructions for execution by the processor1030. Depending on the type of device, the mobile device 990 stores andruns a mobile operating system through which specific applications,including wallpaper programming 945, are executed. Applications, such asthe wallpaper programming 945, may be a native application, a hybridapplication, or a web application (e.g., a dynamic web page executed bya web browser) that runs on mobile device 990 to generate sets ofwallpaper images 968A-K, 969A-K, . . . 975A-K based on one or both of:(i) the spatial user input selection 978, and (ii) the time user inputselection 979. Examples of mobile operating systems include GoogleAndroid, Apple iOS (I-Phone or iPad devices), Windows Mobile, AmazonFire OS, RIM BlackBerry operating system, or the like.

It will be understood that the mobile device 990 is just one type ofhost computer in the wallpaper system 900 and that other arrangementsmay be utilized. For example, a server system 998, such as that shown inFIG. 9, may generate the sets of wallpaper images 968A-K, 969A-K, . . .975A-K after generation of the raw images 858A-B, via thedepth-capturing camera of the eyewear device 100.

FIG. 11 is a flowchart of a method with steps that can be implemented inthe wallpaper system 900 to apply to the original video 964 to createand present a wallpaper image 968A-K, 969A-K, . . . 975A-K. Beginning inblock 1100, the method includes capturing, via a depth-capturing camera,a sequence of original images 965A-H of an original video 964. Each ofthe original images 965A-H is associated with a respective timecoordinate 966A-H on a time (T) axis for a presentation time in theoriginal video 964.

Proceeding now to block 1110, the method further includes presenting,via an image display 180A-B, 1080, an original image 965A-H of theoriginal video 964 to a user. Continuing to block 1120, the methodfurther includes receiving, via a user input device 991, 1091 from theuser one or both of: (i) a spatial user input selection 978 to apply tothe original video 964, and (ii) a time user input selection 979 toapply to the original video 964.

In a first example, the step of receiving, via the user input device991, 1091 from the user one or both of: (i) the spatial user inputselection 978, and (ii) the time user input selection 979 to apply tothe original video 964 includes the following steps. First, receiving onan input surface of a touch sensor at least one finger contact inputtedfrom the user. Second, tracking, via a sensing circuit, the at least onefinger contact on the input surface. Third, detecting one or both of:(i) the spatial user input selection 978, and (ii) the time user inputselection 979 to apply to the original video 964 on the input surface ofthe touch sensor based on the at least one finger contact from the user.For example, the time user input selection 979 is detected by receivinga vertical scrolling (swiping) type of finger gesture on the touchsensor.

In a second example, the step of receiving, via the user input device991, 1091 from the user one or both of: (i) the spatial user inputselection 978, and (ii) the time user input selection 979 to apply tothe original video 964 includes first, tracking, via a movement tracker981, movement of a mobile device 990. Movement tracking is achieved by:(i) measuring, via at least one accelerometer of the movement tracker981, the acceleration of the mobile device 990, (ii) measuring, via atleast one gyroscope of the movement tracker 981, the rotation of themobile device 990, or (iii) measuring, via an inertial measurement unitof the movement tracker 981, both the acceleration and the rotation ofthe mobile device 990. Second, detecting one or both of: (i) the spatialuser input selection 978, and (ii) the time user input selection 979 toapply to the original video 964 by detecting at least one variation ofthe tracked movement over a time period. For example, the spatial userinput selection 978 is detected by sensing horizontal tilting of themobile device 990 via the movement tracker 981 that includes an inertialmeasurement unit 972.

In response to receiving one or both of: (i) the spatial user inputselection 978, and (ii) the time user input selection 979, blocks 1130and 1140 are executed. Moving to block 1130, the method further includesapplying one or both of: (i) a respective spatial movement parameter976A-K associated with the spatial user input selection 978, and (ii)the respective time coordinate 966A-H associated with the time userinput selection 979 to the original video 964. Finishing now in block1140, the method further includes presenting, via the image display180A-B, 1080, a wallpaper image associated with one or both of: (i) therespective spatial movement parameter 976A-K, and (ii) the respectivetime coordinate 966A-H.

In the 2D image processing example, the wallpaper image is generatedfrom the original image 965A-H by the following steps. First,calculating: (i) a left image disparity map 961A between a left pixelmatrix of pixels and a right pixel matrix of pixels, and (ii) a rightimage disparity map 961B between the right pixel matrix and the leftpixel matrix. Second, determining the respective spatial movementparameter 976A-K of the left pixel matrix and the right pixel matrixalong at least one of: (i) an X axis for horizontal position movement,and (ii) a Y axis for vertical position movement. Third, filling up aleft interpolated pixel matrix by moving pixels in the left pixel matrixalong at least one of: (i) the X axis, and (ii) the Y axis based on therespective spatial movement parameter 976A-K. Fourth, filling up a rightinterpolated pixel matrix by moving pixels in the right pixel matrixalong at least one of: (i) the X axis, and (ii) the Y axis based on therespective spatial movement parameter 976A-K. Fifth, blending togetherthe left interpolated pixel matrix and the right interpolated pixelmatrix to create the wallpaper image.

For example, once two disparity maps are created (one left imagedisparity map 961A and one right image disparity map 961B), the spatialmovement parameter 976A-K moves between 0 and 1 to set or skew thespatial movement or rotation of the generated wallpaper image. Supposethe spatial movement parameter 976A-K skews horizontally and when set to0.0 (e.g., 976A) skews to the left image completely and horizontalmovement parameter 966 set to 1.0 (e.g., 976K) skews to the right imagecompletely. If the spatial movement parameter 976A is set to 0.0, thenthe weight is set to output the left image. For example, wallpaperimages 968A, 969A, and 975A are generated for original images 965A,965B, and 965H, respectively. If the spatial movement parameter 976K isset to 1.0, then the weight is set to output the right image as thewallpaper image 968K. For example, wallpaper images 968K, 969K, and 975Kare generated for original images 965A, 965B, and 965H, respectively.When spatial movement parameter 976B-J is not equal to 0.0 or 1.0 (atintermediate values), the spatial movement or rotation is somewhat inbetween the left and right images. Setting the spatial movementparameter 976B-J to 0.1-0.9, fills up empty interpolated pixel matrices967A-B with RGB values to derive intermediate wallpaper images 968B-J,969B-J, . . . 975B-J, respectively from the original images 965A-H. Forexample, setting the spatial movement parameter 976F to 0.5, the pixelsin the left image are moved halfway to the corresponding pixel in theright image according to the respective disparity value from the leftimage disparity map 961A. For example, the respective disparity valuefrom the left image disparity map 961A is multiplied by 0.5 and added tothe X axis location coordinate to derive the left moved X axis locationcoordinate 968A. The right interpolated pixel matrix is filled up in thesame manner by moving the pixels in the right image halfway to thecorresponding pixel in the left image according to the respectivedisparity value from the right image disparity map 961B. For example,the respective disparity value from the right image disparity map 961Bis multiplied by 0.5 and added to the X axis location coordinate toderive the right moved X axis location coordinate. So, for each pixel,the color value stays the same, but the X axis location coordinate ismoved on the X axis by half of the disparity value. If a pixel has novalue (occluded), but neighbor pixels have values, a pixel value iscalculated for the occluded pixel based on the weighted neighbor pixelstogether with a disparity confidence level. In this example, setting thespatial movement parameter 976F to 0.5, creates a field of viewmimicking each of the original images 965A-H.

In another example, assume the spatial movement parameter 976B is set to0.1 To fill up the left interpolated pixel matrix the followingcalculation is used: for each left pixel in the left image, therespective disparity value from the left image disparity map 961A ismultiplied by 0.1 to derive the respective left moved X axis locationcoordinate. To fill up the right interpolated pixel matrix the followingcalculation is used: for each right pixel in the right image, therespective disparity value from the right image disparity map 961B ismultiplied by 0.9 derive the respective right moved X axis locationcoordinate. This creates a novel view in between the left and rightimages.

The step of generating the wallpaper image 968A-K, 969A-K, . . . 975A-Kis achieved by blending together the left interpolated pixel matrix andthe right interpolated pixel matrix. This blending is based on disparityconfidence levels (e.g., by weighing contributions of each side),gradients, or combination thereof in the left image disparity map 961Aand the right image disparity map 961B. The disparity confidence levelvalue is based, for instance, on the magnitude of correlation betweenthe left and the right pixels. Although one might expect to obtain thesame image, the wallpaper image is not the same due to reflection,illumination, etc. being different from the varying perspectives in theleft image and the right image (hence, the term light field effects).This creates the wallpaper images 968A-K with the novel fields of view.

In the 3D image processing example, the wallpaper image 968A-K, 969A-K,975A-K is generated by the following steps. First, creating, via thedepth-capturing camera, a respective depth image 962A-H corresponding tothe original image 965A-H. The respective depth image 962A-H is formedof a respective mesh of vertices 963A-H. Each vertex representing apixel in a three-dimensional scene. Each vertex has a positionattribute. The position attribute of each vertex is based on athree-dimensional location coordinate system and includes an X locationcoordinate on an X axis for horizontal position, a Y location coordinateon a Y axis for vertical position, and a Z location coordinate on a Zaxis for depth (distance). Each vertex further includes one or more of acolor attribute, a texture attribute, or a reflectance attribute.Second, generating, the presented wallpaper image 968A-K, 969A-K, 975A-Kfrom the original image by rotating the respective depth image 962A-Hbased on the respective spatial movement parameter 976A-K. Therespective spatial movement parameter 976A-K is along at least one of:(i) the X axis for horizontal position movement, (ii) the Y axis forvertical position movement, and (iii) the Z axis for depth (distance)movement.

FIGS. 12-22 demonstrate user interaction with an original video 964 of aman wearing white shorts and a t-shirt. In the original video 964, theman holds and swings around a swimming noodle through space and time atdifferent time coordinates 966A-H like a ninja, which is captured aseight original images 965A-H. The wallpaper video 967 is generated andpresented based on the original video 964 and includes eight respectivesets of wallpaper images 968A-K, 969A-K, 975A-K (with eleven differentviewpoints) that responsively pan through space around the eightoriginal images 965A-H based on the various spatial user inputselections 978A-C of FIGS. 13-15 and forward and rewind in time based onthe various time user input selections 979A-F of FIGS. 17-22.

During initialization, the original video 964 is set as wallpaper on amobile device 990. FIG. 12 illustrates an example of a first originalimage 965A associated with a first time coordinate 966A of the originalvideo 964. First original image 965A is a processed (e.g., rectified)image captured by one or both of the visible light cameras 114A-B.

In FIGS. 13-15, three spatial user input selections 978A-C arerespectively received via a movement tracker 981 (e.g., IMU) type ofuser input device 1091 as horizontal left and right tilting for movementof the video along the x-axis 1205. FIG. 13 illustrates receiving, via amovement tracker 981, a first spatial user input selection 978A (e.g.,horizontal tilting to the left) to manipulate the first original image965A by a first spatial movement parameter 976A that is associated withthe leftmost field of view (e.g., 0.0). As shown, the image display 1080of the mobile device 990 responsively presents a first wallpaper image968A associated with the first spatial movement parameter 976A. Thefirst wallpaper image 968A is extracted from the first set of wallpaperimages 968A-K that is generated for the first original image 965A.

FIG. 14 illustrates receiving, via the movement tracker 981, a secondspatial user input selection 978B (e.g., horizontal tilting to theright) to manipulate the first original image 965A by a second spatialmovement parameter 976B that is associated with a left intermediatefield of view (e.g., 0.2). As shown, the image display 1080 of themobile device 990 responsively presents a second wallpaper image 968Bassociated with the second spatial movement parameter 976B. The secondwallpaper image 968B is extracted from the first set of wallpaper images968A-K that is generated for the first original image 965A.

FIG. 15 illustrates receiving, via the movement tracker 981, a thirdspatial user input selection 978C (e.g., horizontal tilting to the left)to again manipulate the first original image 965A by the first spatialmovement parameter 976A that is associated with the leftmost field ofview (e.g., 0.0) as previously depicted in FIG. 13. As shown, the imagedisplay 1080 of the mobile device 990 again responsively presents thefirst wallpaper image 968A associated with the first spatial movementparameter 976A.

In FIGS. 16-22, six time user input selections 979A-F are respectivelyreceived via a touch screen sensor type of user input device 1091 asupwards and downwards vertical scrolling for movement of the video alongthe y-axis 1210. FIG. 16 again illustrates the first wallpaper image968A of FIG. 15 and depicts initiating of finger contact 1600, via atouch sensor type of user input device 1091, to manipulate a timecoordinate 966A-H (e.g., temporal coordinate). This is accomplished byreceiving a time user input selection (e.g., 979A-H).

FIG. 17 illustrates receiving, via the touch sensor type of user inputdevice 1091, a first time user input selection 979A (e.g., verticalscrolling upwards). As shown, the image display 1080 of the mobiledevice 990 responsively presents a first wallpaper image 969C associatedwith a second time coordinate 966B. The first wallpaper image 969C isextracted from the second set of wallpaper images 969A-K that isgenerated for the second original image 965B.

FIG. 18 illustrates receiving, via the touch sensor type of user inputdevice 1091, a second time user input selection 979B (e.g., verticalscrolling upwards). As shown, the image display 1080 of the mobiledevice 990 responsively presents a first wallpaper image 970C associatedwith a third time coordinate 966C. The first wallpaper image 970C isextracted from the third set of wallpaper images 970A-K that isgenerated for the third original image 965C.

FIG. 19 illustrates receiving, via the touch sensor type of user inputdevice 1091, a third time user input selection 979C (e.g., verticalscrolling upwards). As shown, the image display 1080 of the mobiledevice 990 responsively presents a first wallpaper image 971C associatedwith a fourth time coordinate 966D. The first wallpaper image 971C isextracted from the fourth set of wallpaper images 971A-K that isgenerated for the fourth original image 965D.

FIG. 20 illustrates receiving, via the touch sensor type of user inputdevice 1091, a fourth time user input selection 979D (e.g., verticalscrolling upwards). As shown, the image display 1080 of the mobiledevice 990 responsively presents a first wallpaper image 972C associatedwith a fifth time coordinate 966E. The first wallpaper image 972C isextracted from the fifth set of wallpaper images 972A-K that isgenerated for the fifth original image 965E.

FIG. 21 illustrates receiving, via the touch sensor type of user inputdevice 1091, a fifth time user input selection 979E (e.g., verticalscrolling downwards). As shown, the image display 1080 of the mobiledevice 990 again responsively presents the first wallpaper image 971Cassociated with the fourth time coordinate 966D of FIG. 19. The firstwallpaper image 971C is extracted from the fourth set of wallpaperimages 971A-K that is generated for the fourth original image 965D.

FIG. 22 illustrates receiving, via the touch sensor type of user inputdevice 1091, a sixth time user input selection 979F (e.g., verticalscrolling downwards). As shown, the image display 1080 of the mobiledevice 990 again responsively presents the first wallpaper image 968Cassociated with the first time coordinate 966A of FIGS. 15-16. The firstwallpaper image 968C is extracted from the first set of wallpaper images968A-K that is generated for the first original image 965A.

FIGS. 23-25 depict the wallpaper selection and setting process by a useron a mobile device 990, in which a single original image 965A of a manwith a beard wearing a red button-up shirt is smiling at a camera is setas wallpaper. FIG. 23 illustrates wallpaper selection 2300 on the mobiledevice 990, including various types of original videos and originalimages that are selectable as wallpaper. FIG. 24 illustrates selectionof the original image 965A as wallpaper on the mobile device 990. FIG.25 illustrates wallpaper setting 2500 and loading of the original image965A of FIG. 24 as wallpaper on the mobile device 990.

FIGS. 26-28 demonstrate user interaction with the single original image965A, in which the man with the beard wearing the red button-up shirt issmiling at the camera. FIG. 26 illustrates the original image 965A ofFIG. 25, which is a processed (e.g., rectified) image captured by one orboth of the visible light cameras 114A-B. In FIGS. 27-28, spatial userinput selections 978A-B are received via the movement tracker 981 andrespective wallpaper images 968A-B are generated. FIG. 27 illustratesreceiving, via the movement tracker 981 type of user input device 1091,a first spatial user input selection 978A (e.g., horizontal tilting tothe left) to manipulate the first original image 965A by a first spatialmovement parameter 976A that is associated with the leftmost field ofview (e.g., 0.0). As shown, the image display 1080 of the mobile device990 responsively presents a first wallpaper image 968A associated withthe first spatial movement parameter 976A. The first wallpaper image968A is generated from the first original image 965A.

FIG. 28 illustrates receiving, via the movement tracker, a secondspatial user input selection 978B (e.g., horizontal tilting to theright) to manipulate the first original image 965A by a second spatialmovement parameter 976B that is associated with a left intermediatefield of view (e.g., 0.2). As shown, the image display 1080 of themobile device 990 responsively presents a second wallpaper image 968Bassociated with the second spatial movement parameter 976B. The secondwallpaper image 968B is generated from the first original image 965A.

FIGS. 29-31 again depict the wallpaper selection and setting process bya user on a mobile device 990, in which an original video 964 of a pinkFrisbee being thrown across a boardwalk bridge is set as wallpaper. FIG.29 illustrates wallpaper selection 2900 on the mobile device 990,including various types of original videos and original images that areselectable as wallpaper. FIG. 30 illustrates selection of the originalvideo 964 (shown as being represented by a second original image 965B)as wallpaper on the mobile device 990. FIG. 31 illustrates wallpapersetting 3100 and loading of the original video 964 of FIG. 30 aswallpaper on the mobile device 990.

FIGS. 32-37 demonstrate user interaction with the original video 964, inwhich the pink Frisbee is being thrown across a boardwalk bridge. FIG.32 illustrates the second original image 965B, which is a processed(e.g., rectified) image captured by one or both of the visible lightcameras 114A-B. Second original image 965B is associated with the secondtime coordinate 966B of the original video 964.

In FIGS. 33-35, three time user input selections 979A-C are respectivelyreceived via a touch screen sensor type of user input device 1091 asupwards and downwards vertical scrolling. FIG. 33 illustrates receiving,via the touch sensor type of user input device 1091, a first time userinput selection 979A (e.g., vertical scrolling downwards). As shown, theimage display 1080 of the mobile device 990 responsively presents asixth wallpaper image 968F associated with a first time coordinate 966Fof the first original image 965A. Comparing FIGS. 32-33, it can be seenthat the pink Frisbee is closer to the camera in FIG. 33 than FIG. 32.Hence, the wallpaper video 967 is moved backwards (rewound) in time inFIG. 33 compared to FIG. 32. Spatially, the sixth wallpaper image 968Fis selected because the sixth spatial movement parameter 976F mostclosely mimics (resembles or maps) the field of view of the presentedsecond original image 965B, which is halfway between the left image andthe right image captured by the left and right visible light cameras114A-B. The spatial movement parameter 976F is set to 0.5 for the sixthwallpaper image 968F. The sixth wallpaper image 968F is extracted fromthe first set of wallpaper images 968A-K that is generated for the firstoriginal image 965A.

FIG. 34 illustrates receiving, via the touch sensor type of user inputdevice 1091, a second time user input selection 979B (e.g., verticalscrolling upwards). As shown, the image display 1080 of the mobiledevice 990 responsively presents a sixth wallpaper image 970F associatedwith a third time coordinate 966C of the third original image 965C.Comparing FIGS. 33-34, it can be seen that the pink Frisbee is furtheraway from the camera in FIG. 34 than FIG. 33. Comparing FIG. 34 withFIG. 32, it can be seen that the pink Frisbee is further away from thecamera in FIG. 34 than FIG. 32. Hence, the wallpaper video 967 is movedforwards in time in FIG. 34 compared to both FIGS. 32-33. The sixthwallpaper image 970F is extracted from the third set of wallpaper images970A-K that is generated for the third original image 965C.

FIG. 35 illustrates receiving, via the touch sensor type of user inputdevice 1091, a third time user input selection 979C (e.g., verticalscrolling upwards). As shown, the image display 1080 of the mobiledevice 990 responsively presents a sixth wallpaper image 971F associatedwith a fourth time coordinate 966D. The sixth wallpaper image 971F isextracted from the fourth set of wallpaper images 971A-K that isgenerated for the fourth original image 965D.

In FIGS. 36-37, two spatial user input selections 978A-B arerespectively received via a movement tracker 981 (e.g., IMU) type ofuser input device 1091 as horizontal left and right tilting. FIG. 36illustrates receiving, via a movement tracker 981, a first spatial userinput selection 978A (e.g., horizontal tilting to the left) tomanipulate the fourth original image 965D (or sixth wallpaper image 971Fof FIG. 35) by a first spatial movement parameter 976A that isassociated with the leftmost field of view (e.g., 0.0). As shown, theimage display 1080 of the mobile device 990 responsively presents afirst wallpaper image 971A associated with the first spatial movementparameter 976A. The first wallpaper image 971A is extracted from thefourth set of wallpaper images 971A-K that is generated for the fourthoriginal image 965D.

FIG. 37 illustrates receiving, via the movement tracker 981, a secondspatial user input selection 978B (e.g., horizontal tilting to theright) to manipulate the fourth original image 965D (or first wallpaperimage 971A of FIG. 36) by an eighth spatial movement parameter 976H thatis associated with a right intermediate field of view (e.g., 0.7). Asshown, the image display 1080 of the mobile device 990 responsivelypresents an eighth wallpaper image 971H associated with the eighthspatial movement parameter 976H. The eighth wallpaper image 971H isextracted from the fourth set of wallpaper images 971A-K that isgenerated for the fourth original image 965D.

Any of the wallpaper effect functionality described herein for theeyewear device 100, mobile device 990, and server system 998 can beembodied in one or more applications as described previously. Accordingto some embodiments, “function,” “functions,” “application,”“applications,” “instruction,” “instructions,” or “programming” areprogram(s) that execute functions defined in the programs. Variousprogramming languages can be employed to create one or more of theapplications, structured in a variety of manners, such asobject-oriented programming languages (e.g., Objective-C, Java, or C++)or procedural programming languages (e.g., C or assembly language). In aspecific example, a third party application (e.g., an applicationdeveloped using the ANDROID™ or IOS™ software development kit (SDK) byan entity other than the vendor of the particular platform) may bemobile software running on a mobile operating system such as IOS™,ANDROID™ WINDOWS® Phone, or another mobile operating systems. In thisexample, the third-party application can invoke API calls provided bythe operating system to facilitate functionality described herein.

Hence, a machine-readable medium may take many forms of tangible storagemedium. Non-volatile storage media include, for example, optical ormagnetic disks, such as any of the storage devices in any computer(s) orthe like, such as may be used to implement the client device, mediagateway, transcoder, etc. shown in the drawings. Volatile storage mediainclude dynamic memory, such as main memory of such a computer platform.Tangible transmission media include coaxial cables; copper wire andfiber optics, including the wires that comprise a bus within a computersystem. Carrier-wave transmission media may take the form of electric orelectromagnetic signals, or acoustic or light waves such as thosegenerated during radio frequency (RF) and infrared (IR) datacommunications. Common forms of computer-readable media thereforeinclude for example: a floppy disk, a flexible disk, hard disk, magnetictape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any otheroptical medium, punch cards paper tape, any other physical storagemedium with patterns of holes, a RAM, a PROM and EPROM, a FLASH-EPROM,any other memory chip or cartridge, a carrier wave transporting data orinstructions, cables or links transporting such a carrier wave, or anyother medium from which a computer may read programming code and/ordata. Many of these forms of computer readable media may be involved incarrying one or more sequences of one or more instructions to aprocessor for execution.

The scope of protection is limited solely by the claims that now follow.That scope is intended and should be interpreted to be as broad as isconsistent with the ordinary meaning of the language that is used in theclaims when interpreted in light of this specification and theprosecution history that follows and to encompass all structural andfunctional equivalents. Notwithstanding, none of the claims are intendedto embrace subject matter that fails to satisfy the requirement ofSections 101, 102, or 103 of the Patent Act, nor should they beinterpreted in such a way. Any unintended embracement of such subjectmatter is hereby disclaimed.

Except as stated immediately above, nothing that has been stated orillustrated is intended or should be interpreted to cause a dedicationof any component, step, feature, object, benefit, advantage, orequivalent to the public, regardless of whether it is or is not recitedin the claims.

It will be understood that the terms and expressions used herein havethe ordinary meaning as is accorded to such terms and expressions withrespect to their corresponding respective areas of inquiry and studyexcept where specific meanings have otherwise been set forth herein.Relational terms such as first and second and the like may be usedsolely to distinguish one entity or action from another withoutnecessarily requiring or implying any actual such relationship or orderbetween such entities or actions. The terms “comprises,” “comprising,”“includes,” “including,” or any other variation thereof, are intended tocover a non-exclusive inclusion, such that a process, method, article,or apparatus that comprises or includes a list of elements or steps doesnot include only those elements or steps but may include other elementsor steps not expressly listed or inherent to such process, method,article, or apparatus. An element preceded by “a” or “an” does not,without further constraints, preclude the existence of additionalidentical elements in the process, method, article, or apparatus thatcomprises the element.

Unless otherwise stated, any and all measurements, values, ratings,positions, magnitudes, sizes, and other specifications that are setforth in this specification, including in the claims that follow, areapproximate, not exact. Such amounts are intended to have a reasonablerange that is consistent with the functions to which they relate andwith what is customary in the art to which they pertain. For example,unless expressly stated otherwise, a parameter value or the like mayvary by as much as ±10% from the stated amount.

In addition, in the foregoing Detailed Description, it can be seen thatvarious features are grouped together in various examples for thepurpose of streamlining the disclosure. This method of disclosure is notto be interpreted as reflecting an intention that the claimed examplesrequire more features than are expressly recited in each claim. Rather,as the following claims reflect, the subject matter to be protected liesin less than all features of any single disclosed example. Thus, thefollowing claims are hereby incorporated into the Detailed Description,with each claim standing on its own as a separately claimed subjectmatter.

While the foregoing has described what are considered to be the bestmode and other examples, it is understood that various modifications maybe made therein and that the subject matter disclosed herein may beimplemented in various forms and examples, and that they may be appliedin numerous applications, only some of which have been described herein.It is intended by the following claims to claim any and allmodifications and variations that fall within the true scope of thepresent concepts.

What is claimed is:
 1. A wallpaper system comprising: an image displayfor presenting a wallpaper video including a sequence of wallpaperimages, wherein: the wallpaper images are two-dimensional (2D) and basedon raw images or processed raw images, each of the wallpaper images isassociated with a respective time coordinate on a time (T) axis for apresentation time and a respective spatial movement parameter of arespective original image of an original video, wherein upon selectionof the original video as wallpaper by the user, a wallpaper image matrixis created and persistently stored in the memory, including respectivesets of wallpaper images of the wallpaper video, such that: eachrespective set of wallpaper images provides an appearance of a spatialmovement or rotation around the respective original image of theoriginal video, and each wallpaper image within the respective set ofwallpaper images corresponds to a different spatial movement parameterwithin the respective original image; an image display driver coupled tothe image display to control the image display to present the wallpapervideo; a user input device to receive from a user: (i) a spatial userinput selection that manipulates a spatial movement parameter of thewallpaper images, and (ii) a time user input selection to apply to thewallpaper video to move the wallpaper video forwards or backwards intime; a memory; a processor coupled to the image display driver, theuser input device, and the memory; and wallpaper programming in thememory, wherein execution of the wallpaper programming by the processorconfigures the wallpaper system to perform functions, includingfunctions to: present, via the image display, a first wallpaper image ofthe wallpaper video to the user; receive, via the user input device,from the user: (i) the spatial user input selection, and (ii) the timeuser input selection to apply to the wallpaper video; and in response toreceiving: (i) the spatial user input selection, and (ii) the time userinput selection: determine: (i) the respective spatial movementparameter associated with the spatial user input selection, and (ii) therespective time coordinate associated with the time user inputselection; and present, via the image display, a second wallpaper imageat an angular orientation determined by the respective spatial movementparameter and at a point in time of the wallpaper video determined bythe respective time coordinate.
 2. The wallpaper system of claim 1,wherein: a mobile device includes the user input device; the user inputdevice includes: a touch sensor including an input surface and a sensorarray that is coupled to the input surface to receive at least onefinger contact inputted from a user; a sensing circuit integrated intoor connected to the touch sensor and connected to the processor, thesensing circuit configured to measure voltage to track the at least onefinger contact on the input surface; the function of receiving, via theuser input device, from the user: (i) the spatial user input selection,and (ii) the time user input selection to apply to the wallpaper videoincludes: receiving on the input surface of the touch sensor the atleast one finger contact inputted from the user; tracking, via thesensing circuit, the at least one finger contact on the input surface;and detecting: (i) the spatial user input selection, and (ii) the timeuser input selection to apply to the wallpaper video on the inputsurface of the touch sensor based on the at least one finger contactfrom the user.
 3. The wallpaper system of claim 2, wherein: the timeuser input selection is detected; and the time user input selectionincludes a vertical swipe type of finger gesture on the touch sensor. 4.The wallpaper system of claim 1, wherein: a mobile device includes theuser input device; the user input device includes a movement trackercoupled to the processor to track movement of the mobile device, themovement tracker including: (i) at least one accelerometer to measureacceleration of the mobile device, (ii) at least one gyroscope tomeasure rotation of the mobile device, or (iii) an inertial measurementunit (IMU) having the at least one accelerometer and the at least onegyroscope; and the function of receiving, via the user input device,from the user: (i) the spatial user input selection, and (ii) the timeuser input selection from the user to apply to the wallpaper videoincludes: tracking, via the movement tracker, movement of the mobiledevice by: (i) measuring, via the at least one accelerometer, theacceleration of the mobile device, (ii) measuring, via the at least onegyroscope, the rotation of the mobile device, or (iii) measuring, viathe inertial measurement unit, both the acceleration and the rotation ofthe mobile device; and detecting: (i) the spatial user input selection,and (ii) the time user input selection to apply to the wallpaper videoby detecting at least one variation of the tracked movement over a timeperiod.
 5. The wallpaper system of claim 4, wherein: the spatial userinput selection is detected; and the spatial user input selectionincludes horizontal tilting of the mobile device.
 6. The wallpapersystem of claim 1, further comprising: an eyewear device including: aframe; a temple connected to a lateral side of the frame; and adepth-capturing camera, wherein the depth-capturing camera is supportedby at least one of the frame or the temple and includes: (i) at leasttwo visible light cameras with overlapping fields of view, or (ii) aleast one visible light camera and a depth sensor.
 7. The wallpapersystem of claim 6, wherein: the depth-capturing camera includes the atleast two visible light cameras with overlapping fields of view; the atleast two visible light cameras include a left visible light camera tocapture a left raw image and a right visible light camera to capture aright raw image; execution of the wallpaper programming by the processorconfigures the wallpaper system to perform functions, includingfunctions to generate, the second wallpaper image by: calculating: (i) aleft image disparity map between a left pixel matrix of pixels and aright pixel matrix of pixels, and (ii) a right image disparity mapbetween the right pixel matrix and the left pixel matrix, such that: theleft raw image or a left processed image include the left pixel matrix,and the right raw image or a right processed image include the rightpixel matrix; determining the respective spatial movement parameter ofthe left pixel matrix and the right pixel matrix along an X axis forhorizontal position movement; filling up a left interpolated pixelmatrix by moving pixels in the left pixel matrix along the X axis basedon the respective spatial movement parameter; filling up a rightinterpolated pixel matrix by moving pixels in the right pixel matrixalong the X axis based on the spatial movement parameter; and blendingtogether the left interpolated pixel matrix and the right interpolatedpixel matrix to create the second wallpaper image.
 8. The wallpapersystem of claim 7, wherein: the right interpolated pixel matrix isfilled up by moving pixels in the right pixel matrix along the X axisfurther based on a complement of the spatial movement parameter.
 9. Thewallpaper system of claim 1, wherein: the depth-capturing cameraincludes the at least two visible light cameras with overlapping fieldsof view; the at least two visible light cameras include a left visiblelight camera to capture a left raw image and a right visible lightcamera to capture a right raw image; the function of creating, thewallpaper image matrix, including the sets of wallpaper images of thewallpaper video includes: capturing, via the depth-capturing camera, theleft raw image and the right raw image corresponding to the respectiveoriginal image; calculating: (i) a left image disparity map between aleft pixel matrix of pixels and a right pixel matrix of pixels, and (ii)a right image disparity map between the right pixel matrix and the leftpixel matrix, such that: the left raw image or a left processed imageinclude the left pixel matrix, and the right raw image or a rightprocessed image include the right pixel matrix; for each wallpaper imageof the respective set of wallpaper images: determining the respectivespatial movement parameter of the left pixel matrix and the right pixelmatrix of the respective original image along an X axis for horizontalposition movement; filling up a left interpolated pixel matrix by movingpixels in the left pixel matrix along the X axis based on the respectivespatial movement parameter of the respective original image; filling upa right interpolated pixel matrix by moving pixels in the right pixelmatrix along at the X axis based on the respective spatial movementparameter of the respective original image; and blending together theleft interpolated pixel matrix and the right interpolated pixel matrixto create the respective wallpaper image.
 10. The wallpaper system ofclaim 6, wherein execution of the wallpaper programming by the processorconfigures the wallpaper system to perform functions, includingfunctions to: create, via the depth-capturing camera, a respective depthimage corresponding to the first wallpaper image, such that: therespective depth image is formed of a respective mesh of vertices, eachvertex representing a pixel in a three-dimensional scene; each vertexhas a position attribute, the position attribute of each vertex is basedon a three-dimensional location coordinate system and includes an Xlocation coordinate on an X axis for horizontal position, a Y locationcoordinate on a Y axis for vertical position, and a Z locationcoordinate on a Z axis for depth; each vertex further includes one ormore of a color attribute, a texture attribute, or a reflectanceattribute; and generate, the presented second wallpaper image by:rotating the respective depth image based on the respective spatialmovement parameter, and the respective spatial movement parameter beingalong at least one of: (i) the X axis for horizontal position movement,(ii) the Y axis for vertical position movement, and (iii) the Z axis fordepth movement.
 11. The wallpaper system of claim 1, wherein: theprocessor comprises a first processor and a second processor; the memorycomprises a first memory and a second memory; the wallpaper programmingcomprises a first wallpaper programming and a second wallpaperprogramming; the eyewear device includes: a first network communicationinterface for communication over a network; the first processor coupledto the first network communication interface; the first memoryaccessible to the first processor; and first wallpaper programming inthe first memory, wherein execution of the first wallpaper programmingby the first processor configures the eyewear device to performfunctions to capture the raw images or the processed raw images; and thewallpaper system further comprises a host computer coupled to theeyewear device over the network, the host computer including: a secondnetwork communication interface for communication over the network; thesecond processor coupled to the second network communication interface;the second memory accessible to the second processor; and secondwallpaper programming in the second memory, wherein execution of thesecond wallpaper programming by the second processor configures the hostcomputer to perform the functions to: present, via the image display,the first wallpaper image of the wallpaper video to the user; receive,via the user input device, from the user: (i) the spatial user inputselection, and (ii) the time user input selection to apply to thewallpaper video; in response to receiving: (i) the spatial user inputselection, and (ii) the time user input selection: determine: (i) therespective spatial movement parameter associated with the spatial userinput selection, and (ii) the respective time coordinate associated withthe time user input selection; and present, via the image display, thesecond wallpaper image associated with: (i) the respective spatialmovement parameter, and (ii) the respective time coordinate.
 12. Thewallpaper system of claim 11, wherein: the host computer is a mobiledevice; the network is a wireless short-range network or a wirelesslocal area network; and the user input device includes a touch screen ora computer mouse.
 13. A method comprising steps of: capturing a sequenceof original images of an original video, wherein the original images aretwo-dimensional (2D) and based on raw images or processed raw images,each of the original images being associated with a respective timecoordinate on a time (T) axis for a presentation time and a respectivespatial movement parameter of a respective original image in theoriginal video, wherein upon selection of an original video as wallpaperby the user, a wallpaper image matrix is created and persistently storedin the memory, including respective sets of wallpaper images of awallpaper video, such that: each respective set of wallpaper imagesprovides an appearance of a spatial movement or rotation around therespective original image of the original video, and each wallpaperimage within the respective set of wallpaper images corresponds to adifferent spatial movement parameter within the respective originalimage; presenting, via an image display, the wallpaper video; receiving,via a user input device, from the user: (i) a spatial user inputselection to manipulate a spatial movement parameter of the wallpaperimages, and (ii) a time user input selection to apply to the wallpapervideo to move the wallpaper video forwards or backwards in time; inresponse to receiving: (i) the spatial user input selection, and (ii)the time user input selection: applying: (i) a respective spatialmovement parameter associated with the spatial user input selection, and(ii) the respective time coordinate associated with the time user inputselection, and presenting, via the image display, a wallpaper image atan angular orientation determined by the respective spatial movementparameter and at a point in time of the original video determined by therespective time coordinate.
 14. The method of claim 13, wherein the stepof receiving, via the user input device, from the user: (i) the spatialuser input selection, and (ii) the time user input selection to apply tothe wallpaper video includes: receiving on an input surface of a touchsensor at least one finger contact inputted from the user; tracking, viaa sensing circuit, the at least one finger contact on the input surface;and detecting: (i) the spatial user input selection, and (ii) the timeuser input selection to apply to the wallpaper video on the inputsurface of the touch sensor based on the at least one finger contactfrom the user.
 15. The method of claim 14, wherein: the time user inputselection is detected; and the time user input includes a vertical swipetype of finger gesture on the touch sensor.
 16. The method of claim 13,wherein the step of receiving, via the user input device, from the user:(i) the spatial user input selection, and (ii) the time user inputselection to apply to the wallpaper video includes: tracking, via amovement tracker, movement of a mobile device by: (i) measuring, via atleast one accelerometer of the movement tracker, the acceleration of themobile device, (ii) measuring, via at least one gyroscope of themovement tracker, the rotation of the mobile device, or (iii) measuring,via an inertial measurement unit of the movement tracker, both theacceleration and the rotation of the mobile device; and detecting: (i)the spatial user input selection, and (ii) the time user input selectionto apply to the wallpaper video by detecting at least one variation ofthe tracked movement over a time period.
 17. The method of claim 16,wherein: the spatial user input selection is detected; and the spatialuser input selection includes horizontal tilting of the mobile device.18. A method comprising steps of: capturing, via a depth-capturingcamera, a sequence of original images of an original video, each of theoriginal images associated with a respective time coordinate on a time(T) axis for a presentation time in the original video; presenting, viaan image display, an original image of the original video to a user;receiving, via a user input device, from the user one or both of: (i) aspatial user input selection to apply to the original video, and (ii) atime user input selection to apply to the original video; in response toreceiving one or both of: (i) the spatial user input selection, and (ii)the time user input selection: applying one or both of: (i) a respectivespatial movement parameter associated with the spatial user inputselection, and (ii) the respective time coordinate associated with thetime user input selection to the original video, generating a wallpaperimage from the original image by: calculating: (i) a left imagedisparity map between a left pixel matrix of pixels and a right pixelmatrix of pixels, and (ii) a right image disparity map between the rightpixel matrix and the left pixel matrix; determining the respectivespatial movement parameter of the left pixel matrix and the right pixelmatrix along at least one of: (i) an X axis for horizontal positionmovement, and (ii) a Y axis for vertical position movement; filling up aleft interpolated pixel matrix by moving pixels in the left pixel matrixalong at least one of: (i) the X axis, and (ii) the Y axis based on therespective spatial movement parameter; filling up a right interpolatedpixel matrix by moving pixels in the right pixel matrix along at leastone of: (i) the X axis, and (ii) the Y axis based on the respectivespatial movement parameter; and blending together the left interpolatedpixel matrix and the right interpolated pixel matrix to create thewallpaper image; and presenting, via the image display, the wallpaperimage associated with one or both of: (i) the respective spatialmovement parameter, and (ii) the respective time coordinate.
 19. Themethod of claim 13, further comprising steps of: creating, via thedepth-capturing camera, a respective depth image corresponding to thewallpaper image, such that: the respective depth image is formed of arespective mesh of vertices, each vertex representing a pixel in athree-dimensional scene; each vertex has a position attribute, theposition attribute of each vertex is based on a three-dimensionallocation coordinate system and includes an X location coordinate on an Xaxis for horizontal position, a Y location coordinate on a Y axis forvertical position, and a Z location coordinate on a Z axis for depth;each vertex further includes one or more of a color attribute, a textureattribute, or a reflectance attribute; and generating, the presentedwallpaper image from an original image by: rotating the respective depthimage based on the respective spatial movement parameter; and therespective spatial movement parameter being along at least one of: (i)the X axis for horizontal position movement, (ii) the Y axis forvertical position movement, and (iii) the Z axis for depth movement.